I'm using AWS DynamoDB to store data in JSON format. The Partition key is "device" and sort key is "timestamp". I can query the table for a specific device in a range of dates. I can then filter the content by the specific endpoint (in the nested "reports" object) the application is interested in.
{
"device": "AAA111",
"attr1": "bbb",
"reports": [
{
"endpoint": 1,
"value": "23"
},
{
"endpoint": 3,
"value": "26"
},
{
"endpoint": 4,
"value": "20"
}
],
.........
............
...........
"timestamp": "2017-11-30T03:50:30z"
}
The problem I have is if for example, I want to retrieve the latest value of an specific "endpoint". So, I can retrieve the latest record for a "device" based on the latest "timestamp", but it doesn't guarantee this record will contain value for this particular endpoint (not all records contains all endpoints). To solve this I have to basically scan the latest records (in descending order) and return the first object where the endpoint is found. Also, I don't know how many records I have to retrieve to find one...
I'm wondering if there is a better way of doing this... I tried with secondary indexes but this would require to duplicate the data, creating an object for each endpoint value (duplicating the common data). I would like to avoid this...
I would appreciate any hints on how to solve this issue.
Thanks
Gus
Related
I have a dynamodb table whose items have below structures.
{
"url": "some-url1",
"dependencies": [
"dependency-1",
"dependency-2",
"dependency-3",
"dependency-4"
],
"status": "active"
}
{
"url": "some-url2",
"dependencies": [
"dependency-2",
],
"status": "inactive"
}
{
"url": "some-url3",
"dependencies": [
"dependency-1",
],
"status": "active"
}
Here, url is defined as the partition key and there is no sort key.
The query which needs to run needs to find all the records with a specific dependency and status.
For example - find all the records for whom dependency-1 is present in dependencies list and whose status is active.
So for the above records, record 1st and 3rd should be returned.
Do I need to set GSI on dependencies or is this something which cannot be done in dynamodb ?
You cannot create a GSI on a nested value. You can however create a GSI on status but you would need to be careful as it has a low cardinality meaning you could limit your throughput to 1000 writes per second if all of your items being written to the table have the same status. Of course if you never intend to scale that high then it's no issue.
Your other option is to use a Scan where you read your entire data set and use a FilterExpression to filter based on dependency and status.
Depending on the SDK you use you can find some example operations here:
https://github.com/aws-samples/aws-dynamodb-examples/tree/master/DynamoDB-SDK-Examples
I have a collection "foo". Each document in the collection has a property/field "bar" that is an array of objects like so:
foo = [{
bar: [{
id: "random_string",
"status": "string"
}, {
id: "random_string",
"status": "string2"
}]
}, {
bar: [{
id: "random_string",
"status": "string"
}, {
id: "random_string",
"status": "string2"
}]
}]
What I want to achieve is I want to be able to query the db so that I can get two different collections, one with all the documents in the "foo" collection that have "string" as a value of at least one of the objects in the "bar" array, and another collection for all the documents that have "string2" as a value of at least one of the objects in the "bar" array.
Is that even possible? I'm struggling quite a lot to this one, so any help would be greatly appreciated. I'm also happy to change the db schema if needed, totally open to suggestions!
You can't do this with a single query because Firestore currently doesn't support logical OR conditions. In other words, you can't have a query that gives you all the documents where any one of a set of conditions is true.
Also, you need to be able to call out a particular field in a document in order to perform a query against it. Without a specific field to use, the query can't use an index to speed things up, and the query would never scale at the magnitude offered by Firestore.
So I am asking here about the concept of binding a table.
Usually it is pretty straight forward just use ngFor for all rows/columns.
However what I would like to do is that, for each cell it should be binded to an object of two attributes:
Cell Content
Header of the column, which is shared between all cells in that column.
In my table I should be able to add a row or a column as I want.
Which is simple, but all cells sharing column_header is the tricky part.
Now this table represent a form, so I can do processing after the user clicks submit and solve the issue.
I am just looking for other smarter ideas.
Thanks in advance.
EDIT| Clarification example
Given the object from this post:
AngularJS - Building a dynamic table based on a json
Object
{
"name": "john"
"colours": [{"id": 1, "name": "green"},{"id": 2, "name": "blue"}]
}
I wanted to be something like that
{
"name": "john",
"colours": [
{
"id": 1,
"column":{
"name": "green",
"header":"H1"
}
},
{
"id": 2,
"column":{
"name": "blue",
"header":"H1"
}
}
]
}
Where the header attribute is shared between both
There are some ways:
iterate your original objects and generate new objects based on your requirement and structure.
in ngFor you can use index, to get the index of the column, then you can use the index to get that column header. you can check the api
I'm wondering if it is possible to get all info about a specific date from Freebase.
I can easily retrieve info about a date giving a specific topic, for example, to grab all persons of interest who were born on a specific date:
[{
"type":"/people/person",
"limit":1000,
"sort":"name",
"name":null,
"guid":null,
"timestamp":null,
"/people/person/date_of_birth":"1955-02-24"
}]
Is it possible to grab all types? I'm after things like people born on that date (which I have), major events (start of a war, assassination of a person of interest, etc), and so on.
Essentially I want to match all fields that are dates and return the full information about that entry, regardless of type.
Reflection is what you need here:
[{
"/type/reflect/any_value": [{
"type": "/type/datetime",
"value": "1955-02-24",
"link": {
"source": {
"id": null
},
"master_property": null
}
}]
}]
A couple of notes on that: the MQL manual I've linked to is somewhat bitrotted in its details but is still the best documentation that exists on MQL. Secondly, there's what I'm pretty sure is in MQL bug if you use "*": null or more specifically "target_value": null in the link clause above which makes it ignore the outer value you specified... so don't do that :-)
I tried Googling and searching everywhere, but couldn't find a definitive authority on this topic. While being true to REST principles, how should I design the HTTP interface for:
An ordered list (get, add, insert into position, reorder, remove)
A set (get, add, remove)
A hash-table (get, add, remove)
NOTE: These data structures are to contain references to existing resources with known ids
That's how I would do it for an ordered list and hash table. I guess the methods would be the same for a set and a list:
Ordered list
Get item 123:
GET /list/123
Append an item to the list:
POST /list/
Insert new item into position 5:
POST /list/?position=5
Move item 123 to position 3:
PUT /list/123?position=3
Delete item 123:
DELETE /list/123
Delete item at position 3:
DELETE /list/?position=3
Of course, your API should update the indexes of all the elements when doing insertion and deletion.
Hash table
Get item "somekey":
GET /hashtable/somekey
Add item "somekey":
POST /hashtable/somekey
Remove item "somekey":
DELETE /hashtable/somekey
#dadads
You can not define such interface directly.
An ordered list (get, add, insert into position, reorder, remove)
By excluding "insert into position" and "reorder" you can perfectly implement "get", "add" and "remove" for example:
You define your resource /service/users
You can use POST /service/users to add new user to the "users" collection
You can GET /service/users to retrieve users
You can GET /service/users/user-id to retrieve particular user
You can DELETE /service/users/user-id from users collection
This is a very rough example, though it outlines some ideas. In order to achieve "reorder" and "insert into position" you need to implement your own action semantics which you can include in your resource representation and let client know HOW to perform these operations. As a reference you can see this JSON PATCH specification proposal: https://www.rfc-editor.org/rfc/rfc6902 which tries to describe such operations.
It is not necessary to use already existing media format, you can define your own under your own namespace for example: application/vnd.your-company.format-name+json which describes these capabilities and also advertises this information to clients.
You should decouple the transport mechanism from the underlying application. I would consider designing the application correctly, then figure out how to access it via HTTP. This way you could easily add or change the transport mechanisms (SOAP, SCA, etc) without affecting the underlying application.
Once you have the application correctly designed, consider accessing it from the HTTP requests via something like an Adapter or Visitor pattern.
This is my idea for reordering.
There is a HTTP method called PATCH that is used to update fragments of a resource. Give your resource a new property called index, then make a call with PATCH method
PATCH /collection
[
{
"id: "original index 0"
"index": 1
}
{
"id: "original index 1"
"index": 0
}
]
Then your server back-end needs to figure out how to do this atomically. But interface-wise, I think this is the best way to stay true to RESTful.
Alternatively, there is a better solution, but it may not apply to everyone's case. Since ordering always depends some sort of criteria, it can even be as simple as insertion order. Let your collection url support an orderBy query string, and let this orderBy dictate on how the result gets ordered. Then during your reordering call from client, just update the resource's property used for the ordering criteria.
I came to this question mostly looking for a RESTful way to reorder. I don't really like any of the answers, so here is what I think is most RESTful.
For reorder you could make the order a resource:
/list/order
Then you can do normal operations on it (for these examples assume a list with 5 items currently in it):
"items":" [
{
"id": "A",
"name": "Monkey"
},
{
"id": "B",
"name": "Cow"
},
{
"id": "C",
"name": "Horse"
},
{
"id": "D",
"name": "Turkey"
},
{
"id": "E",
"name": "Tasmanian Devil"
},
]
Note that "order" is not included in the resource response. It's not needed - the order is implicitly specified by the response order of the items.
GET /list/order
returns a list of item ids in their correct order
['A','B','C','D','E']
POST /list/order
with payload ['D','B','C','A','E']
GET /list/order
returns a list of item ids in their correct order
['D','B','C','A','E']
Also obviously you would return the items in the list in the correct order when you do a GET on /list.
GET /list
returns a list of items in their correct order
"items":" [
{
"id": "D",
"name": "Turkey"
},
{
"id": "B",
"name": "Cow"
},
{
"id": "C",
"name": "Horse"
},
{
"id": "A",
"name": "Monkey"
},
{
"id": "E",
"name": "Tasmanian Devil"
},
]