The following query in Datastore return the expected number of results:
SELECT timestamp from leadgenie_campaign_model_dev where campaign = "3667f39d-a3ff-4acb-b1ca-6f730bbc7989"
This query is backed by an index, which allows for the projection.
But this one doesn't return any results, despite the fact that an index also exists. attrs is a JSON string
SELECT timestamp, attrs from leadgenie_campaign_model_dev where campaign = "3667f39d-a3ff-4acb-b1ca-6f730bbc7989"
Here's the spec for the indexes:
indexes:
- kind: leadgenie_campaign_model_dev
properties:
- name: campaign
- name: attrs
- name: timestamp
- kind: leadgenie_campaign_model_dev
properties:
- name: campaign
- name: timestamp
- name: attrs
- kind: leadgenie_campaign_model_dev
properties:
- name: campaign
- name: timestamp
Is your attrs field more than 1,500 bytes by any chance? That is a limit for a string property to be indexed. See https://cloud.google.com/datastore/docs/concepts/limits .
The solution was to include the subproperty in the query, after creating an index for those properties specifically.
SELECT timestamp, attrs.score, attrs.sz from leadgenie_campaign_model_dev where campaign = "3667f39d-a3ff-4acb-b1ca-6f730bbc7989"
google.api_core.exceptions.FailedPrecondition: 400 no matching index found. recommended index is:
- kind: leadgenie_campaign_model_dev
properties:
- name: campaign
- name: attrs.score
- name: attrs.sz
- name: timestamp
Related
I make some yamls by openapi3.0 as follow:
- name: ip
in: query
schema:
type: string
oneOf:
- format: ipv4
- format: ipv6
- name: ipv4
oneOf:
- $ref: 'xxx/Ipv4Type'
- type: array
items:
$ref: 'xxx/Ipv4'
But for some reason, I must convert it to swagger2.0. How should I translate them up here?
As per title, does anybody know how to configure kafka topics using dapr pubsub component? Specifically I would like to configure retention.ms topic property.
Current pubsub.yaml:
apiVersion: dapr.io/v1alpha1
kind: Component
metadata:
name: pubsub
namespace: default
spec:
type: pubsub.kafka
version: v1
metadata:
# Kafka broker connection setting
- name: brokers
value: "kafka:9092"
- name: authRequired
value: "false"
- name: initialOffset
value: "oldest"
- name: maxMessageBytes
value: 8192
Thanks in advance.
im looking for a way to trigger a workflow using webhook json payload key for example
if payload is {"action":"copy","id":"someid"}
on triggers
triggers:
template:
- "group1"
"use this template if action key is 'copy'"
- "group2"
"use this template" if action key is 'move'"
i have created a example sensor file but, argo sersor is complaining that eventSourceName used multiple times.
apiVersion: argoproj.io/v1alpha1
kind: Sensor
metadata:
name: testing-sensor
spec:
template:
serviceAccountName: argo
dependencies:
- name: data
eventSourceName: testhook
eventName: test
filters:
data:
- path: body.message
type: string
value:
- copy
- name: data1
eventSourceName: testhook
eventName: test
filters:
data:
- path: body.message
type: string
value:
- move
triggers:
- template:
conditions: "data"
name: group-1
- template:
conditions: "data1"
name: group-2
If some one could help it would be nice
You should use a data filter in your Sensor. (Example sensor.)
It would look something like this:
apiVersion: argoproj.io/v1alpha1
kind: Sensor
metadata:
name: data-filter
spec:
dependencies:
- name: test-dep
eventSourceName: webhook
eventName: example
filters:
data:
- path: body.action
type: string
value:
- "copy"
triggers:
- template:
name: data-workflow
k8s:
group: argoproj.io
version: v1alpha1
resource: workflows
operation: create
# ...
I have a DynamoDB table in which I store link data (url, date, category, tags etc).
I need to be able to -
lookup items by url (check url doesn't already exist)
find all items stored after a given date
Based on the above I set up the schema with url as primary hash key and secondary index on date, as follows -
AWSTemplateFormatVersion: '2010-09-09'
Parameters:
TableName:
Type: String
Default: "my_links"
HashAttr:
Type: String
Default: url
IndexAttr:
Type: String
Default: date
ReadCapacity:
Type: Number
Default: 5
WriteCapacity:
Type: Number
Default: 5
Resources:
Table:
Properties:
KeySchema:
- AttributeName: !Ref HashAttr
KeyType: HASH
AttributeDefinitions:
- AttributeName: !Ref HashAttr
AttributeType: S
- AttributeName: !Ref IndexAttr
AttributeType: S
GlobalSecondaryIndexes:
- IndexName: !Ref IndexAttr
KeySchema:
- AttributeName: !Ref IndexAttr
KeyType: HASH
Projection:
ProjectionType: ALL
ProvisionedThroughput:
ReadCapacityUnits: !Ref ReadCapacity
WriteCapacityUnits: !Ref WriteCapacity
ProvisionedThroughput:
ReadCapacityUnits: !Ref ReadCapacity
WriteCapacityUnits: !Ref WriteCapacity
TableName: !Ref TableName
Type: AWS::DynamoDB::Table
I can query the table by date as follows, but only using an eq condition -
ddb=boto3.resource("dynamodb")
table=ddb.Table("my_links")
from boto3.dynamodb.conditions import Key
queryexp=Key('date').eq("2020-02-19")
for item in table.query(IndexName="date",
KeyConditionExpression=queryexp)["Items"]:
print (item)
If I use a gte in place of eq condition I get the following -
botocore.exceptions.ClientError: An error occurred (ValidationException) when calling the Query operation: Query key condition not supported
I can however query the table using a scan and a gte condition -
filterexp=Key('date').gte("2020-02-19")
for item in table.scan(FilterExpression=filterexp)["Items"]:
print (item)
But then I'm guessing I don't need the secondary index any more and also that this will get very expensive as the table get large :-/
So I'd rather stick with a secondary index and query if possible (am I thinking about this right ?), but what do I need to do to the schema to be able to fetch all items after a date ?
You can't use GTE.
Queries support EQ | LE | LT | GE | GT | BEGINS_WITH | BETWEEN
Check this article https://medium.com/cloud-native-the-gathering/querying-dynamodb-by-date-range-899b751a6ef2
So this was useful -
https://stackoverflow.com/a/38790120/124179
and also this -
How to query DynamoDB by date (range key), with no obvious hash key?
In the end the answer was to denormalised the data, replacing the date field (and associated secondary index) with a week field, then searching for one or more specific weeks using eq query condition and joining the results (I only need a couple of weeks data)
Obviously could replace week with month for increased range but reduced granularity.
Getting the error cannot do operation on a non-existent table after running sls offline start and attempting to access the users endpoint. serverless.yml file is as follows:
service:
name: digital-secret
plugins:
- serverless-dynamodb-local
- serverless-offline # must be last in the list
custom:
userTableName: 'users-table-${self:provider.stage}'
dynamoDb:
start:
migrate: true
provider:
name: aws
runtime: nodejs8.10
stage: dev
region: us-east-2
iamRoleStatements:
- Effect: Allow
Action:
- 'dynamodb:Query'
- 'dynamodb:Scan'
- 'dynamodb:GetItem'
- 'dynamodb:PutItem'
- 'dynamodb:UpdateItem'
- 'dynamodb:DeleteItem'
Resource:
- { "Fn::GetAtt": ["usersTable", "Arn"] }
environment:
USERS_TABLE: ${self:custom.userTableName}
functions:
app:
handler: index.handler
events:
- http: ANY /
- http: 'ANY {proxy+}'
user:
handler: index.handler
events:
- http: 'GET /users/{proxy+}'
- http: 'POST /users'
resources:
Resources:
usersTable:
Type: 'AWS::DynamoDB::Table'
Properties:
TableName: ${self:custom.userTableName}
AttributeDefinitions:
- AttributeName: userId
AttributeType: S
KeySchema:
- AttributeName: userId
KeyType: HASH
ProvisionedThroughput:
ReadCapacityUnits: 1
WriteCapacityUnits: 1
Can anyone help point out what is wrong here? I've looked through the docs and at many different examples available online, but nothing I can see is different than the above.
The serverless-dynamodb-local docs say that the custom block should be structured like this:
custom:
dynamodb:
start:
migrate: true
You have dynamoDb instead of dynamodb
If anyone else is having issues with this, I spent hours trying to track down this issue and it was because I inadvertently had the wrong case for the [r]esources section in serverless.yml
Resources: <-- Needs to be lower case 'r'
Resources:
usersTable:
Type: 'AWS::DynamoDB::Table'
Properties:
...