I'm currently doing some tests on google datastore, but I'm having a problem with my queries.
If I believe in the documentation https://cloud.google.com/datastore/docs/concepts/queries we can realize a filter on several columns with the instruction EQUALS.
But when testing, I get an error from the API.
While searching on Datastore's github, I found this reference: https://github.com/GoogleCloudPlatform/google-cloud-dotnet/issues/304 which corresponds to my problem, except that for my case the query to the look good.
Here is the request sent:
{
{
"kind": [{
"name": "talk.message"
}],
"filter": {
"compositeFilter": {
"op": "AND",
"filters": [{
"propertyFilter": {
"property": {
"name": "Conversation"
},
"op": "EQUAL",
"value": {
"stringValue": "2f16c14f6939464ea687d316438ad4cb"
}
}
},
{
"propertyFilter": {
"property": {
"name": "CreatedOn"
},
"op": "LESS_THAN_OR_EQUAL",
"value": {
"timestampValue": "2019-03-15T10:43:31.474166300Z"
}
}
},
{
"propertyFilter": {
"property": {
"name": "CreatedOn"
},
"op": "GREATER_THAN_OR_EQUAL",
"value": {
"timestampValue": "2019-03-14T10:43:31.474175100Z"
}
}
}
]
}
}
}
}
And here is the answer from the API:
{Grpc.Core.RpcException: Status(
StatusCode=FailedPrecondition,
Detail="no matching index found. recommended index is:
- kind: talk.message
properties:
- name: Conversation
- name: CreatedOn"
)
According to the documentation, this should be good... but it's not !
What am I missing ?
Your query includes both an EQUALS (on Conversation) and a non-EQUALS filter (on CreatedOn), therefore you need a composite index to fulfil the query. So your query is valid, but it needs a composite index to be able to run the query.
Related
We have custom dimension define in Google Analytics Data API v1Beta for extracting data from Google Analytics GA4 account.
This worked fine till yesterday.
Error:
Please remove customEvent:institution_id to make the request compatible. The request's dimensions & metrics are incompatible. To learn more, see https://ga-dev-tools.web.app/ga4/dimensions-metrics-explorer/
POST - https://analyticsdata.googleapis.com/v1beta/properties/{propertyId}:runReport
Body
{
"dateRanges": [
{
"startDate": "2022-08-29",
"endDate": "2022-12-07"
}
],
"dimensions": [
{
"name": "customEvent:institution_id"
},
{
"name": "pagePathPlusQueryString"
}
],
"dimensionFilter": {
"andGroup": {
"expressions": [
{
"filter": {
"fieldName": "customEvent:institution_id",
"inListFilter": {
"values": [
"47339486-a1e5-47be-abce-e4270af23rte"
]
}
}
},
{
"filter": {
"fieldName": "pagePathPlusQueryString",
"stringFilter": {
"matchType": "PARTIAL_REGEXP",
"value": "/clip/.+",
"caseSensitive": false
}
}
}
]
}
},
"metrics": [
{
"name": "screenPageViews"
}
],
"metricAggregations": [
"TOTAL"
],
"limit": "10000"
}
Dimensions that include the query string like pagePathPlusQueryString are only compatible with a limited set of dimensions & metrics.
This changed was announced 2022-09-13 Schema compatibility changes announcement.
It went live earlier this week. So the cause if the error is that the "customEvent:institution_id" dimension is not compatible with "pagePathPlusQueryString".
This is my http POST requst body...
{
"structuredQuery": {
"select": {
"fields": [
{
"fieldPath": "name"
},
{
"fieldPath": "taxId"
},
{
"fieldPath": "mailingAddress"
}
]
},
"from": [
{
"collectionId": "orgs"
}
],
"where": {
"fieldFilter": {
"field": {
"fieldPath": "orgId"
},
"op": "IN",
"value": {
"arrayValue": {
"values": [
{
"stringValue": ""
},
{
"nullValue": null
}
]
}
}
}
}
}
It fails to return orgs where the orgId field is completely missing from the document. It correctly includes orgs where the orgId field is present and equal to empty string. This is accessing a Cloud Firestore db.
Due to the way Firestore indexes data, it is not possible to query for documents for which a certain field "is completely missing from the document": the field needs to exist in order for the Firestore index to take it into account. More details on the indexing mechanism in the following official video.
You may store an empty value in this field, as you mention in your question.
I am querying firestore using query
https://firestore.googleapis.com/v1/projects/myproject/databases/(default)/documents/mycollection
I am getting following Json. Can someone please help me how can I filter my query based on Rate field. I am writing following query and it doesn't work
https://firestore.googleapis.com/v1/projects/myproject/databases/(default)/documents/mycolletion?Rate="15"
{
"documents": [
{
"name": "0C45nDuozgQDOwEx5xHR",
"fields": {
"Clinic": {
"stringValue": "American Hospital"
},
"Rate": {
"stringValue": "140"
},
,`enter code here`
"createTime": "2020-06-28T20:32:18.776123Z",
"updateTime": "2020-07-22T21:19:24.061647Z"
},
{
"name": "Jm3tNVWmk4Q1pk87KL1m",
"fields": {
"Clinic": {
"stringValue": "Cleaveland clinic"
},
"Rate": {
"stringValue": "150"
},
"createTime": "2020-06-28T20:28:03.726819Z",
"updateTime": "2020-07-22T21:19:05.073019Z"
}
}
The problem is that the Rate field is an object (inside another object to make it worse), so in order to achieve that you would either need to update you Firestore structure to do it all in the request URL, or you would have to use a structured query in the body of the request.
Change the structure:
In order to work with the request you already have, you will need to change the structure to (using the 1st document as an example):
{
"name": "0C45nDuozgQDOwEx5xHR",
"Clinic": "American Hospital",
"Rate": "140",
"createTime": "2020-06-28T20:32:18.776123Z",
"updateTime": "2020-07-22T21:19:24.061647Z"
}
which in my opinion makes it simpler although I don't have the full picture
Have a Structured query in your request body:
To keep the structure you already have you will need to use thisURL:
https://firestore.googleapis.com/v1/projects/myproject/databases/(default)/documents/:runQuery
And add this to the body of the request:
"structuredQuery": {
"from": [{
"collectionId": "mycollection"
}],
"where": {
"fieldFilter": {
"fields": {
"Rate": {
"stringValue": "15"
}
}
}
}
}
You can check more details by checking the runQuery Documentation and the structuredQuery Documentation for what else you can do with these options.
Based on the Google Discovery document, and RPC reference, it appears that the :write resource should be available for Firestore database interactions, but performing such a request to my (POST https://firestore.googleapis.com/v1beta1/projects/[my project]/databases/(default)/documents:write) results in:
[
{
"error": {
"code": 400,
"message": "Invalid value (Object), ",
"status": "INVALID_ARGUMENT",
"details": [
{
"#type": "type.googleapis.com/google.rpc.BadRequest",
"fieldViolations": [
{
"description": "Invalid value (Object), "
}
]
}
]
}
}
]
Is this possible? A related SO answer alludes to this being available as a means of field transforms, the same reason I require this, but I cannot construct valid a JSON body to succeed in the request. Currently, variations on the following don't work as expected when trying a minimum successful response:
{
"writes": [
{
"update": {
"name": "projects/{projectId}/databases/[my project]/documents/exampleId",
"fields": {
"example": {
"integerValue": 100
},
"timestamp": {
"nullValue": null
}
},
"transform": {
"document": "projects/[my project]]/databases/(default)/documents/examples/exampleId",
"fieldTransforms": [
{
"fieldPath": "timestamp",
"setToServerValue": "REQUEST_TIME"
}
]
}
}
}
]
}
First of all, note that you should use the v1 version of the REST API, not the betas.
To create a document, you would use the createDocument method, while to update a document you would use the patch one.
For the document creation you should therefore make a POST HTTP Request to the following URL
https://firestore.googleapis.com/v1/projects/<your-project-id>/databases/(default)/documents/<the-desired-collection>
with the following Request body:
{
fields: {
example: {
integerValue: 100
}
}
}
You need to use documents:commit instead of documents:write
also the name field should be in this format:
"name": "projects/projectID/databases/(default)/documents/collectionName/DocumentId"
See this post.
This might be a silly question but I could not manage to filter elasticsearch indexes by a datetime field. I must be missing something.
This is the mapping:
"created_at": {
"type": "date",
"format": "strict_date_optional_time||epoch_millis"
},
This is what I got:
{
"_index": "myindex",
"_type": "myindextype",
"_id": "21c",
"_score": 1,
"_source": {
"code": "21c",
"name": "hello",
...
"created_at": "2015-04-30T13:10:50.107769Z"
}
},
With this query:
"query": {
"filtered": {
"query": {},
"filter": {
"range": {
"created_at": {
"gte": "2015-05-02T13:10:50.107769Z"
"format": "strict_date_optional_time||epoch_millis"
}}}}}
I would expect to filter out the entry above. But it returns nothing.
Is there a problem with time format? Because it is directly coming from Django Rest Framework's serializers. They claim that it is ISO 8601 format and elasticsearch claims the same.
I would also like to filter them out by a simpler date like "2015-05-02".
I am stuck. Thank you in advance.
Edit: It does not matter whatever i write into the range filter. It always return all the entries.
This worked. I tried a lot of different things and lost my way at some point.
{
"query": {
"filtered": {
"filter": {
"range": {
"created_at": {
"gte": "2015-05-02"
}
}
}
}
}
}