API Gateway as DynamoDB non-proxy integration with Integration Response for variable Map returns - amazon-dynamodb

Given an AWS Service integration (DynamoDB) in API Gateway, I'm wondering if there's a way to dynamically parse DynamoDB-structured JSON and return it in standard JSON format WITHOUT a standalone Lambda function mapper? For easier demonstration, here's an example of two possible data elements existing in my DynamoDB table I'd like to return in standard JSON format using a single Integration Response mapping template, if possible:
Ex. 1
{
"id": "1",
"desc": "here's a string type at depth 1",
"info": {
"key1": "a string at depth 2"
}
}
Ex. 2
{
"id": "2",
"desc": "here's a string type at depth 1",
"info": {
"key1": "a string at depth 2"
"key2": {
"subkey1": "look, a string at depth 3"
}
}
}
Based on these two examples, we can see these are nested data structures that share top-level keys, but have a variable number of nested tiers.
I've noted the following in all of the answers to questions regarding Integration Mapping Templates WITHOUT using a separate Lambda function to resolve the mapping:
Parsing Integration Response mapping templates for "flat" DynamoDB data -- e.g. something like this, where there's effectively a data depth of 1 since all elements in the data are basic types like S or N:
{
"param1": "some stuff",
"param2": "other stuff"
}
Any answers dealing with more complex data types like M presume the question writer knows without a doubt exactly what the DynamoDB table elements look like. So, they'll say something like 'create the mapping template for each of the layers with the appropriate, expected return type for that field.' Using Ex. 1 to illustrate a solution Integration Response mapping template:
#set($inputRoot = $input.path('$'))
{
"id": "$input.Item.id.S",
"desc": "$input.Item.desc.S",
"info": {
"key1": "$input.Item.info.M.key1.S"
}
}
Now, this works if we actually know what the expected DynamoDB result fields will be for every depth and how many tiers each table element has -- i.e., the data in a NoSQL table should be structured? Barring there being some version of the AppSync VTL resolver $util.dynamodb.toJSON method that can effectively unwrap dynamically-shaped DynamoDB returns, is there a way to do this directly without having to utilize a standalone Lambda function? Is there a way to generalize the response mapping template to account for variably-nested M type data?
One idea I've had is to loop through all of the keys and sending the M types to a secondary loop, but this seems impractical given:
the number of iterations that could theoretically be required to unwrap a deeply-nested object (DynamoDB can support up to 32 layers, last I checked), and
the variability in the nesting from element to element in the table.
To point 1, #foreach has a hard limit on the number of iterations it's capped at: 1000 https://forums.aws.amazon.com/thread.jspa?threadID=225222
I'm dubious as to the prospect of a non-Lambda solution, but still curious. Thoughts?

Related

Drupal 8 jsonapi: How to change the structure of returned json relationships included array?

I am sending a request to the jsonapi node endpoint, and using the following parameter to include the "relationship" objects of those nodes for user (post author data) and user picture (avatar):
include=uid,uid.user_picture
This returns to me the json data of the nodes as well as all of the relationship objects lumped together (regardless of type) into the "included" array. The included array ends up looking like this by default:
"included": [
{
"type": "user--user",
"id": "4c717273-f903-4ffa-abe2-b5e5709ad727",
"attributes": {
"display_name": "cherylm1234",
...etc...
}
},
{
"type": "file--file",
"id": "4c717273-f903-4ffa-abe2-b5e5709ad727",
"attributes": {
"filename": "Cheryl-Fun.jpg",
...etc...
}
}
]
Context: We are trying to integrate jsonapi with an iOS app as the client/consumer.
My Problem: It's difficult to write up data models in the iOS client for these relationships (without looping and sorting every response) since they are all lumped into the same level of the "included" array. Also a problem is that each "type" will have its own set of attributes/fields so data models in iOS need to be built based on "type".
Question: Is it possible to change this structure so that all included objects are sorted by type?
I would like for the "included" array to be indexed by "type" where all objects of that type would be inside that index. Maybe something like:
included['user--user'] = [
{user-object},
{user-object},
...etc.
],
included['file--file'] = [
{file-object},
{file-object},
....etc.
],
I'm assuming this would require a custom module with some sort of hook where I could loop and sort the data before returning it to the iOS app. But, haven't been able to find any documentation on this type of json response modification.
Any help would be much appreciated.
Drupal implements the JSON:API specification. The specification specifies how a resource should be encoded. Both if it is returned as primary data or included as a relation resource.
If you want the data to be returned in another format, you would need to implement your own REST API in Drupal following another specification.
But if this is only about managing the data in a client, I would recommend to use one of the existing libraries for JSON:API specification. A list of implementations is provided at jsonapi.org/implementations/. The JSONAPI Swift package listed on that page seems to be well maintained.

What would be a good API format for “IS NOT SET” clause?

I have a query API against my service, that looks like this (JSON-ish format):
{
filter: {
,attribute2: [val21, val22]
,attribute3: []
}
}
means effectively, select data WHERE attribute2 in ("val21", "val22") AND attribute3 IS NOT NULL in SQL-ish syntax (meaning, the object being returned has attribute 3 set, but I really don't care what its value is. SQL isn't very good at expressing this of course, as my data is key-value store where a key may be "not set" at all instead of being null valued).
I need to expand this API to be able to express IS NOT SET predicate, and I'm at a loss as to what a good way to do so would be.
The only thing I can possibly think of is to add a special "NOT_SET" value in the request API, that would produce NOT SET semantics; but it seems really klunky and hard to grasp:
The API syntax can be thought of as JSON as far as its expressiveness/capability
An ideal answer would reference some well accepted rules on API design, to show that it's "good".
{
filter: {
,attribute2: [val21, val22]
,attribute4: [__NOT_SET__]
}
}
My suggestion would be to move away from trying to use a key-value pair to represent a predicate phrase. You should have a lot more flexibility with a structure similar to:
{
filters: [
{ attribute: "attribute2", verb: "IN", values: [val21, val22] },
{ attribute: "attribute2", verb: "NOT IN", values: [val21, val22] },
{ attribute: "attribute4", verb: "IS NOT SET" },
]
}
You'd want an enum of verbs, of course, and values would have to be optional. You can add more verbs later if you need them, and you're no longer putting quite so much pressure on the poor :. You can also provide to the client a list of supported verbs and how many values (if any) of what type they take, so the client can build the UI dynamically, if desired.
Of course, this is a breaking change, which may or may not be an issue.

Watson Conversation: condition matching input to context array

Taking the car dashboard example, I altered the initial #genre node to be #genre:classical. I also added a list to the contex
"choices":["Beethoven","Mahler 9","Brahms 3rd"]
and the Watson response is "I have 3 selections". The condition on the next node is $choices.contains(input.text). The "Found a match" response is just for testing. It looks like this:
When I test this in the api tool and type "Beethoven" both "Found a match" and "Great choice!..." appear. Same for the other two choices, but only if I type the exact choice, e.g., "Mahler 9". Typing "Mahler" or "mahler" doesn't get a match. I read through the SpEL documentation but couldn't see a way in a one-line condition to parse through the list looking for partial matches.
So my question is, is there an condition expression that would match partial user input, e.g., "Mahler"? I'll be using the Java SDK to code the app server, so alternatively I wondered if I could add a temporary #entity just for this sequence instead of using the context list then delete it when the conversation is done? Or is there a way to construct a more complex condition in the MessageRequest and will Watson recognize it? Or is this just not the right way to go about this? Any pointers, examples or docs much appreciated.
So my question is, is there an condition expression that would match partial user input
You can't add temporary entities or intents. As adding them forces Watson to start training itself (even if you could it through code).
You can however create quite complex regular expressions, pass them in as a context variable.
For example your advanced node can have:
{
"output": {
"text": "Please ask me a question."
},
"context": {
"rx": "fish|[0-9]+"
}
}
Then in you condition you would write.
input.text.matches(context.rx)
This will then trigger if the person mentions a number, or the word fish. So you can create your partial user input checking that way.

Is there any way I can get a list of all possible responses from the google vision api?

I am using the google cloud vision api to analyze pictures. Is there a list of all the possible responses for the labelAnnotations method?
The API reference of Vision API gives an overview of all the possible JSON responses for the different image annotation requests.
The labelAnnotation request returns a generic EntityAnnotation response, you can find the JSON representation here, also containing more information about the JSON representation of BoundingPoly, LocationInfo and Property:
{
"mid": string,
"locale": string,
"description": string,
"score": number,
"confidence": number,
"topicality": number,
"boundingPoly": {
object(BoundingPoly)
},
"locations": [
{
object(LocationInfo)
}
],
"properties": [
{
object(Property)
}
],
}
I think you're asking whether you can get a look at the list of possible labels/entities that the Cloud Vision API will detect. If that's the case, the short answer is no, not in any manageable way.
The more complicated answer is sort of, since most labels will have a property for the knowledge graph entry (e.g., {desc: 'dog', mid: '/m/0bt9lr'}). This means that you can look-up more information about the label/entity using the Knowledge Graph API.
While you can't "store a copy" of Google's Knowledge Graph as a list of choices in a drop-down on a page, you can use the API to do a look-up after the Vision API responds with an ID.

Currently Using MySQL, Looking at DocumentDB

I currently use MySQL, after looking into Document DB it seems like it may be a good move. I do a TON (95%) of querying for single records. As my database gets larger, the time its taking to do this seems to be getting slower. Both reading and writing. I'm curious based on the (simplified) scheme below if it could be a good move to a DocumentDB, and what the layout would be for said schema (i'm a bit new to documentDB)
User
UserID
Username
CreatedDate
Tank
TankID
UserID REF User.UserID
TankName
Awards
Map
MapID
MapName
MapFIle
MapData
MapID REF Map.MapID
TankID REF Tank.TankID
Rank
Color
TimePlayed
Equipment
Everytime a player joins, the data from Tank,MapaData is Queried to gather a full tank object. Every time they die, win an award, kill somebody, or exit the game, the data is then written back out to tank,and mapdata.
The website queries the User table for login, which stores the username and a hash of the password. Once logged in the users are able to modify/delete/create new tanks on the website, which inserts records into the tank/mapdata tables.
The website also stores Top 25 in the World, t25 in map, t25 for each color, t25 for each color for each map.
That's about the only query patterns I can think of at this moment.
Based on the provided information you have the choice of several schema designs (with JSON as examples). I've made some assumptions, such as that more than one tank can be on one map and map data is only linked to a single map. You have to tweak it for your needs. I also try to provide some advantages and disadvantages of every solution.
Option #1 (Single collection)
This should be the easiest, but not the best solution. Here you put everything into one document with extreme "denormalization".
{
"mapname": "map1",
"mapfile": "mapfile1",
"data": {
"rank": "rank1",
"color": "color1",
...
"tanks": [
{
"name": "tank1",
...
"user": {
"name": "user1",
...
}
},
{
...
}
]
}
}
This solution works best when you do a lot of writes, rare updates and reads where you want to get all information together. On the other side it has a lot of disadvantages, such as storing user information directly into your application data (an example would be the password hash).
Option #2 (Two collections)
Put your user data into one collection and the other data into a second collection.
User collection
{
"id": 1,
"username": "user1",
"password": "passwordhash",
...
}
Data collection
{
"mapname": "map1",
"mapfile": "mapfile1",
"data": {
"rank": "rank1",
"color": "color1",
...
"tanks": [
{
"name": "tank1",
...
"user": userId
}
},
{
...
}
]
}
}
This option is a lot better than the first one. First you don't want to have sensitive user data (such as the hash of the password) in a collection with your other data. Also this works better for reads of the user object, because you just retrieve the information you need without skipping a lot of not needed fields. A disadvantage is that heavy write operations on the tank object can become a problem.
Option #3 (Three collections)
The next step could be to move the tanks out of the data collection into their own collection.
User collection
{
"id": 1,
"username": "user1",
"password": "passwordhash",
...
}
Tank collection
{
"name": "tank1",
...
"user": userId
}
Data collection
{
"mapname": "map1",
"mapfile": "mapfile1",
"data": {
"rank": "rank1",
"color": "color1",
...
"tanks": [
idOfTank1,
idOfTank2,
...
]
}
}
This works best for a lot of writes of single objects, such as the tanks, and reading tanks from their collection. This solution has its problems when reading a lot of data together, for example if you want to get a map and all the tanks in that map. In that case you have to resolve the dependencies of the tanks and the map data.
Summary
As seen, schema design is not easy in a document-oriented database. This is the reason why I asked for the query patterns. To come up with a good design you have to know most of the query patterns in advance. To get started, you should create a simple prototype with a design you think makes sense and test your query patterns with some test data. If that works, you can make minor changes to get even better performance. If not, rethink your query patterns and how a better design could look like. Keep in mind that you don't need a full-blown application for that. Most of that can be tested before a single line of code is written, for example with the administration shell of MongoDB or a simple console application in the case of DocumentDB.

Resources