I am learning how to model a NoSQL databases and trying to understand how to make this works properly.
For my use case I have spin up a local instance of aws dynamo db and connected successfully.
Now, let's say I have two contexts "Users" and "Groups", where one user can belong to any group, and one group can have multiple users. I am trying to avoid multiple collections but I cannot see the way to.
What I have in mind is something like:
{
name: "Bob Smith",
dob: "1980-01-01
groups: [
{
name: "Sports"
CreatedAt: "2022-01-05"
}.
{
name: "Arts"
CreatedAt: "2022-01-08"
}
]
}
What I am not sure is what should we do if I need to create or update a Group? if I follow the previous pattern, how would be the query for the groups?
Shall I have 2 collections?
Thanks guys for your advice.
You would typically only hold a pointer to the groups within a users item, if you need to update the group details then you would do so in the item for the group.
pk
sk
groups
other
user123
USER#user123
groups[{pk:group1, sk: GROUP#group1}, {pk:group3, sk: GROUP#group3}]
user data
user009
USER#user009
groups[{pk:group5, sk: GROUP#group5}, {pk:group7, sk: GROUP#group7}]
user data
group1
GROUP#group1
group-info thats editable
group3
GROUP#group3
group-info thats editable
group5
GROUP#group5
group-info thats editable
group7
GROUP#group7
group-info thats editable
Of course this is not the only way to do this, but with all things NoSQL it totally depends on your access patterns. For example, if you have a requirement to get all users for a given group then you would need to change the above schema:
pk
sk
other
other
user123
USER#user123
user-data
user123
GROUP#group1
some data
user123
GROUP#group3
some data
group1
GROUP#group1
group data
group3
GROUP#group3
group data
Now you can get all the groups a user belongs to by issuing a Query stating:
pk=user123 AND sk BEGINS_WITH(GROUP#).
Then use a Global Secondary Index to get all the users which belong to a given group, with sk being your GSI's partition key:
gsi_pk = GROUP#group1
Related
I am currently new to AWS DynamoDB and noSql.
I am lost here on filtering list value in DynamoDB.
Let's say that I have 2 items in this table.
[
{
"id": 1,
"title": "Robots in Music"
"topics": ["Robots", "Violin"]
},
{
"id": 2,
"title": "Where are good places to see stars"
"topics": ["Robots", "Stars"]
},
]
I want to filter using topics column.
ex.
User wants to get the item with topics having "Robots".
User gets item with id 1 and 2
User wants to get item with topics having "Stars".
Then user gets item with id 2.
I tried to search internet and found that I can use 'QueryFilter' 'contains'.
However, I know that 'contains' is scanning all the table and for DynamoDB they can extract 1MB of data in single query. Which means the action needs to be repeated and it would cost way more than using single index.
Is there any way to use GSI and filter the list effectively?
Unfortunately you cannot index a list type or any other type of nested attrubute, and your use case would require you to Scan the entire table to know which users contained a particular topic.
Would require a Scan
A GetItem if the user wants just id1 or of the user wants both id1 and id2 then a BatchGetItem
Same as 1
GetItem
Of your use-case requires searching nested attributes then you can consider using a relational database or something more flexible like OpenSearch.
I am testing out DynamoDB for a serverless app I am building. I have successfully modeled all of my application's query patterns except one. I was hoping someone could provide some guidance. Here are the details:
Data Model
There are three simple entities: User (~1K records), Product (~100K), ActionItem (~100/product).
A User has a many-to-many relationship with Product.
A Product has a one-to-many relationship with ActionItem.
The Workflow
There's no concept of "Team" for this app. Instead, a user is assigned a set of products which they (and others) are responsible for managing. The user picks the oldest items from their products' action item list, services the item and then closes it.
The use case I am trying to model is: As a user, show me all action items for products to which I am assigned.
Any help would be greatly appreciated.
Really only two options...
If you can store the list of products within the 400KB limit of DDB record, then you could have a record like so...
Hash Key: userID
Sort KEY: "ASSIGNED_PRODUCTS"
Otherwise,
Hash key: UserID
Sort key: "#PRODUCT#10001-54502"
userID in the above might be the raw userid, or if using a GSI, might be something like "#USER#user-id"
I am confused about what Firestore counts toward its composite index limit of 200.
For example, let's say I have thousands of users, and each user has thousands of characters. I want each user to be able to sort through his characters, based on different parameters about those characters. I have the following structure, where {} indicates a wildcard:
users:
{user_ID}
user_name: {user_name}
user_email: {user_email}
characters:
{characterUID}
name: {name}
strength: {strength}
speed: {speed}
stamina: {stamina}
date_created: {timestamp}
So, in android, I would then do this:
Query query = fsDB.collection("users").document("user_ID").collection("characters")
.orderBy("strength").orderBy("name");
Or this:
Query query = fsDB.collection("users").document("user_ID").collection("characters")
.orderBy("speed").orderBy("date_created");
Then, I would create the composite index in Firebase Console as:
Collection Group: characters
Fields Indexed: strength , name
and
Collection Group: characters
Fields Indexed: speed, date_created.
So, is this just 2 composite indexes according to Firebase? Or is this multiplied by the number of Users I have?
If it is multiplied by the number of users, how should I restructure my data so that I do not run into this problem?
Thanks -
Jeff
After discussing with Sam Stern at Firebase,
The answer is that the indexes are not multiplied by the number of users.
From Sam ----
Indexes actually on depend on the collection name so all of the "characters" subcollections of your "users" documents can share the same indexes.
You'd only approach the 200 index limit if you had 200+ different field combinations you wanted to index. Hope that makes sense!
Sam
i have two set of entities in my firebase realtime schema. Called Orders and customers.
so far i was not actually relating them in my app but was just showing them related. the current schema looked like:
{
"orders" : [
{"id" : 1, "name": "abc", "price": 200, "customer": "vik"}
],
"customers" : [
{"cust_id" : "10", "name" : "vik", "type": "existing"}
]
}
so i have a orders list page showing all the orders in a table which i get firing /orders.json
But practically, instead of having the customer name directly in the orders i should have cust_id attribute as that is the key.
That naturally makes it a standard relational schema where i will be free to change customer attributes without worrying about mismatch in orders.
However, the downside i see right away is that if i have say 20 orders to show in the order list table then instead of 1 i will end up firing 21 rest calls (1 to get order list and 20 to fetch customer name for each of the order)
What are the recommendations or standards around this ?
Firebase is a NoSQL database. So the rules of normalization that you know from relational databases don't necessarily apply.
For example: having the customer name in each order is actually quite normal. It saves having to do a client-side join for each customer record, significantly simplifying the code and improving the speed of the operation. But of course it comes at the cost of having to store data multiple times (quite normal in NoSQL databases), and having to consider if/how you update the duplicated data in case of updates of the customer record.
I recommend reading NoSQL data modeling, watching Firebase for SQL developers, and reading my answer on keeping denormalized data up to date.
I have a strong background in relational databases. However, I'm always looking to improve my skills. Recently, I've been exposed to Firebase. It seems pretty interesting. However, I'm slightly confused by the "schema" if that's even the correct term.
From what I can tell, each Firebase "app" basically represents a single "table". Thus, if I am building a web application that has two related, but seperate entities, I would have to have two firebase "apps". For example, perhaps I am building a web application that has football teams, coaches and players. In a relational database, I may have something like this:
Relational Database
Team Coach TeamCoachLookup Player TeamPlayerLookup
---- ----- --------------- ------ ----------------
ID ID ID ID ID
Name FirstName TeamID FirstName TeamID
Location LastName CoachID LastName PlayerID
The above shows a possible relational database structure. Some may want to have a Person table with a RoleID to represent whether the person is a Player or a Coach. That's one approach. Still, when I look at the Firebase model, I have trouble getting my head around how the above would be structured. Would it be:
http://teams.firebaseio.com
http://coaches.firebaseio.com
http://players.firebaseio.com
Where the JSON of each item would represent a row in the database? Or, should it just be http://teams.firebaseio.com and the schema would look like this:
{
coaches: [
{ id:1, firstName:'Joe', lastName:'Smith' }
],
players: [
{ id:1, firstName:'Bill', lastName:'Mans' },
{ id:2, firstName:'Zack', lastName:'Dude' }
]
}
The second approach seems to make more sense to me. However, I do not see how Firebase supports that. Instead, in my mind, it looks like Firebase has one URL for each "table" and the JSON isn't really hierarchical. Am I way off? Is there any documentation that anyone can recommend to me?
Thanks!
The corresponding concepts should be (Firebase <=> relational):
application <=> schema
root node <=> table
child node <=> row
node key <=> row id (typically push ids)
In your concrete example:
football-app.firebaseio.com
teams
fx7Q7q
name: "Foo"
coaches
ix0GWF
firstName: "Joe"
lastName: "Smith"
players
uQ8fJK
firstName: "Bill"
lastName: "Mans"
teamCoachLookup
QkW9uH
team: "fx7Q7q"
coach: "ix0GWF"
teamPlayerLookup
BmI48N
team: "fx7Q7q"
player: "uQ8fJK"
See also https://www.firebase.com/docs/web/guide/structuring-data.html.