I am trying to build a step function that has a choice state based on a map in a result of a dynamo db map. An example result from my dynamo GetItem request would be.
{
"Item": {
"organisationId": {
"S": "Andys-test"
},
"id": {
"S": "Andy2"
},
"states": {
"L": [
{
"M": {
"year": {
"N": "2021"
},
"status": {
"S": "started"
}
}
},
{
"M": {
"year": {
"N": "2022"
},
"status": {
"S": "started"
}
}
}
]
},
},
My condition will be checking the status of the states map against the year 2021. I have attempted to use this JSONPath which from what I can tell is valid, although I am getting nothing in the data flow simulator in the step functions console. I have tried various iterations of the below, with quotes escaped quotes etc and can't get anything to parse the correct value out.
I have been doing this in the input selector as I can see that the result path does not support the [?()] notation.
$.Item.states.L..M[?(#.year.N == 2022) ]
Without an example choice state with some rules, it'll be hard to answer definitively but I think I'm following.
JSONPath with expression filters
I run into this sort of problem more often than I'd like when using JSONPath filter expressions and have to add "helper" tasks to get things moving (wasting valuable state transitions 😣 ).
When you specify a Path that includes a filter expression, the result is always going to be a list (and you won't be able to reference index afterwards either).
ChoiceRules don't really have a comparator that deals with arrays/lists (at least i haven't been able to get it to work, so let me know if you do 😄).
The hack I've found easiest to reason about/maintain is creating a simple Pass Task that "pops" the values I need from the filter expression and then pass that along to the Choice Task.
Here's an example from one of my previous answers (the pattern used in the PassDef task would be defined before your choice task in your scenario, instead of after)
I've found it easier to start with https://jsonpath.herokuapp.com then progress to the data flow simulator. Just remember to always keep in mind whether
you're trying implement a Path or a Reference Path!
Related
https://docs.aws.amazon.com/appsync/latest/devguide/resolver-mapping-template-reference-dynamodb.html#aws-appsync-resolver-mapping-template-reference-dynamodb-query
AppSync doc says that expressionNames and expressionValues are optional fields, but they are always populated by code generation. First question, should they be included as a best practice when working with DynamoDB? If so, why?
AppSync resolver for a query on the partition key:
{
"version": "2017-02-28",
"operation": "Query",
"query": {
"expression": "#partitionKey = :partitionKey",
"expressionNames": {
"#partitionKey": "partitionKey"
},
"expressionValues": {
":partitionKey": {
"S": "${ctx.args.partitionKey}"
}
}
}
}
Second question, what exactly is the layman translation of the expression field here in the code above? What exactly is that statement telling DynamoDB to do? What is the use of the # in "expression": "#partitionKey = :partitionKey" and are the expression names and values just formatting safeguards?
Let me answer your second question first:
expressionNames
expressionNames are used for interpolation. What this means is after interpolation, this filter expression object:
"expression": "#partitionKey = :value",
"expressionNames": {
"#partitionKey": "id"
}
will be transformed to:
"expression": "id = :value",
the #partitionKey acts as a placeholder for your column name id. '#' happens to be the delimiter.
But why?
expressionNames are necessary because certain keywords are reserved by DynamoDB, meaning you can't use these words inside a DynamoDB expression.
expressionValues
When you need to compare anything in a DynamoDB expression, you will need also to use a substitute for the actual value using a placeholder, because the DynamoDB typed value is a complex object.
In the following example:
"expression": "myKey = :partitionKey",
"expressionValues": {
":partitionKey": {
"S": "123"
}
}
:partitionKey is the placeholder for the complex value
{
"S": "123"
}
':' is the different delimiter that tells DynamoDB to use the expressionValues map when replacing.
Why are expressionNames and expressionValues always used by code generation?
It is just simpler for the code generation logic to always use expressionNames and expressionValues because there is no need to have two code paths for reserved/non-reserved DynamoDB words. Using expressionNames will always prevent collisions!
I have a Players collection and a Games collection. I want to construct a data structure that looks as follows:
{
"_id": "1234",
"accounts": {
"battlenet": "blah#1234"
},
"games": {
"overwatch": {
"class": "Hanzo",
"timePlayed": ISODate
},
"world-of-warcraft": {
"class": "Shaman",
"timePlayed": ISODate
}
}
}
games is an object, where every key refers to a specific document in the Games collection's slug attribute. Every value is a sub-schema definition with autoValues.
I can't find any good way to create validation in such a way that it updates an autoform correctly without weird coersion of data. Is there any way to accomplish this validation with simple schema?
I want to create a dashboard which shows information about a limited set of request values :
request:("/path1" OR "/path2" OR "/path3")
What I've tried so far:
I can add filters to the dashboard by clicking on a part of a pie chart, but all these filters are applied as AND filters and not OR. This way of working also requires actual data for all possible request values. Which is not always the case in a test environment.
in Discover I created a saved search but I don't know how I can apply this to my Dashboard so it gets part of the dashboard definition.
Is their a way to do this using the Dashboard editor or does it require some json scripting via Settings->Objects->Dashboards ? If so can you point me a good reference to this (escaped) syntax ?
In Kibana 3 you could define filters of type "either". Does this functionality exist in Kibana 4 ?
I'm using Kibana 4.0.2
I am not sure if this is an answer to your actual question, I'll write it anyway as someone may benefit and I found examples on the Kibana filter syntax to be elusive when googling.
I am trying to define a boolean filter instead of a boolean query in my Discover tab, to unclutter the search field and fascilitate further filtering on a limited set of values.
I found this link to the documentation where AND, OR, NOT filter syntax is described. After a bit of experimenting this was what worked for me, example:
I have a field named host containing the name of the server shipping the log entry. There are quite a few servers, each belonging to one of several redundancy groups. To filter only for log entries produced by the servers "SERVER06 OR SERVER07 OR SERVER08" which happen to belong to a distinct redundancy group B-Servers I can make an OR filter like so:
{
"bool": {
"should": [
{
"query": {
"match": {
"host": {
"query": "SERVER06",
"type": "phrase"
}
}
}
},
{
"query": {
"match": {
"host": {
"query": "SERVER07",
"type": "phrase"
}
}
}
},
{
"query": {
"match": {
"host": {
"query": "SERVER08",
"type": "phrase"
}
}
}
}
]
}
}
and save it as a search called B-Servers. Now I get a filtered list, where I can cherry pick a server with a further and more restrictive filter. Before I had all servers and the quick count only listed the five top entries, so I had to pick one and then edit the filter manually if my target wasn't in the list.
This should be useful for other string type fields too. The documentation should have included a couple of more examples I think, to set the context for the placement of the bool statement and not just a demonstration of the principle.
This link was also useful for demonstrating how to do booleans from the search field rather than as a filter.
[EDIT] An update for Kibana 5.2 as I could not get the previous syntax to work. The following did the trick with 5.2, I used this link to figure it out:
{
"query": {
"bool": {
"should": [
{
"match": {
"host": "SERVER06"
}
},
{
"match": {
"host": "SERVER07"
}
},
{
"match": {
"host": "SERVER08"
}
}
],
"minimum_should_match": 1
}
}
}
Kibana 4 is a total rewrite and apparently not all Kibana 3 features are yet implemented. I've found an "enhancement" ticket in the Kibana github:
https://github.com/elastic/kibana/issues/3693
This closes my question for the moment.
Definitely you can add OR filters in your dashboard. As dashboard is created from saved visualizations, In your visualization you can add filter containing OR which will reflect such data.
As per my understanding of your question I am posting my answer (Feel free to correct me):-
Clicking on pie chart under visualize tab (Create a new visualization).
Selecting a search source - From a new search
Go to Split Slices, select aggregation as filters.
Under Query1 enter the filter you want to apply such as request:("/path1" OR "/path2")
Click on add Filter and add Query2 as: request:("/path2" OR "/path3")
Click on Apply to view the changed pie chart as per filters.
Save Visualization by selecting it from toolbar (2nd option beside search bar).
Go to Dashboard & Click on Add Dashboard & select your saved visualization which will reflect your pie chart.
Hope it answers your question :)
The lucene syntax is "OR" (uppercase), so "foo OR bar".
Once your query is corrected, you can save the search in the Discover tab and refer to that saved search in your Visualization.
You can also manually set the query in the visualization if you don't want the overhead of saving it separately.
In the Firebase security rules file, could one effectively use the '.validate' directives to synchronize schema, that is, make sure when one part of the Firebase is modified, another part gets updated as well? Or is this a bad/invalid idea?
Basically what I'm trying to do is create a Todo list app. I want to be able to create tasks in my todo list with specified tags. For each tag in the app, there should be a respective tag child field in the "tags" field in my Firebase backend. The tag field should contain an ordered list of references to tasks in the "tasks" field. So, if a task is created with "#foo" tag, then it's name (id) should be referenced in "tasks/foo". Also, whenever a task is removed, it's reference in each tag should be removed, and when a tag in has no children it should be removed.
To clarify here's an example of the schema:
{
"tasks": {
"-sdjfaowneono": {
"headline": "Get some milk",
"completed": false,
"tags": {
"0": "all",
"1": "shopping"
}
},
"-asdfhsadfsafwirn": {
"headline": "Pick up the kids",
"completed": false,
"tags": {
"0": "all"
}
}
},
"tags": {
"all": {
"0": "-sdjfaowneono",
"1": "-asdfhsadfsafwirn"
},
"shopping": {
"0": "-sdjfaowneono"
}
}
}
Again, if task "-sdjfaowneono" was removed, so would the "shopping" tag be removed.
In conclusion, I need to synchronize the "tags" schema with the "tasks" schema in a specific way. Should I do this synchronization in the Security Rules? Or should I do this at the app level? And if I do it at the app level, there is a risk that the app bugs out and doesn't add/remove a tag which it should, so how do I overcome that?
To use this with security rules, you will need to change one or the other to use the tag id as the key, rather than an array value, since there is no way to specify a wild card or search multiple children to see if one contains the correct value.
So if you changed your "tags" path to look like this:
"tags": {
"all": {
"-sdjfaowneono": true,
"-asdfhsadfsafwirn": true
},
"shopping": {
"-sdjfaowneono": true
}
}
Then you could validate like this:
"tasks": {
"$task": {
"tags": {
"$tag": {
".validate": "root.child('tags/'+newData.val()+'/'+$task).exists()"
}
}
}
}
This uncovers a basic design practice that I've discovered while building with Firebase: I tend to prefer keys to values for storing any sort of foreign key reference.
Also, keep in mind that this only enforces that they match up on write. It does help with deletions since .validate will only run if the value exists.
You could enforce the same behavior on deletions by appending this to the write rule:
".write": " ... && (newData.exists() || !root.child('tags/'+newData.val()+'/'+$task).exists())"
I have a little problem, maybe can you help me. I'm trying to get the "currency features" from freebase. So I tried to do : "/base/schemastaging/person_extra/net_worth": null but, I can't get the value written on freebase (for example, with Madonna, it's 650,000,000). Do you know, why it's not working ?
First of all, as the property path suggests, /base/schemastaging/person_extra/net_worth is just being staged right now so the final property ID will be something else (follow the mailing list to discuss new schema). You should NOT be using this property for anything other than experimentation.
The reason why you don't see the data that you want withe the following query is because this property is a CVT.
{
"id": "/en/madonna",
"type": "/base/schemastaging/person_extra",
"net_worth": null
}
CVT values are complex objects that need to be expanded to access the values that you want. In this case, net_worth is a CVT so that we can record a person's net worth at different points in time.
If you expand your query to include the relevant properties from /measurement_unit/dated_money_value you'll see the data that you're after.
{
"id": "/en/madonna",
"type": "/base/schemastaging/person_extra",
"net_worth": {
"amount": null,
"currency": null,
"valid_date": null
}
}
One other issue, that isn't obvious from this example, is that since there can be multiple dated money values, you'll need to make your query more precise so as to get only the latest value. You can do that like this;
{
"id": "/en/madonna",
"type": "/base/schemastaging/person_extra",
"net_worth": {
"amount": null,
"currency": null,
"valid_date": null,
"sort": "-valid_date",
"limit": 1,
"optional": true
}
}​
Update: Made net worth an optional property value.