I am trying to filter list of maps from a dynamodb table which is of the following format.
{
id: "Number",
users: {
{ userEmail: abc#gmail.com, age:"23" },
{ userEmail: de#gmail.com, age:"41" }
}
}
I need to get the data of the user with userEmail as "abc#gmail.com". Currently I am doing it using the following dynamodb query. Is there any another efficient way to solve this issue ?
var params = {
TableName: 'users',
Key:{
'id': id
}
};
var docClient = new AWS.DynamoDB.DocumentClient();
docClient.get(params, function (err, data) {
if (!err) {
const users = data.Item.users;
const user = users.filter(function (user) {
return user.email == userEmail;
});
// filtered has the required user in it
});
The only way you can get a single item in dynamo by id if you have a table with a partition key. So you need to have a table that looks like:
Email (string) - partition key
Id (some-type) - user id
...other relevant user data
Unfortunately, since a nested field cannot be a partition key you will have to maintain a separate table here and won't be able to use an index in DynamoDB (neither LSI, nor GSI).
It's a common pattern in NoSQL to duplicate data, so there is nothing unusual in it. If you were using Java, you could use transactions library, to ensure that both tables are in sync.
If you are not going to use Java you could read DynamoDB stream of the original database (where emails are nested fields) and update the new table (where emails are partition keys) when an original table is updated.
Related
I am learning Flutter with Firebase. I have some dummy data in Firebase Realtime Data base as:
{
"postsnode": {
"posts": [
{
"postId": "u1",
"postName": "p1"
}
]
},
"usersnode": {
"users": [
{
"userId": "u1",
"userName": "bla bla 1"
},
{
"userId": "u2",
"userName": "bla bla 2"
}
]
}
}
The screen shot of the console database structure is:
I have successfully performed a query on my usersnode object to get the specific users:
void queryDB(BuildContext context) async {
AppUtil.showLoader(context: context);
FirebaseDatabase.instance.ref().child('usersnode/users').orderByChild('userId').equalTo("u1").get().then((snapshot) {
if (snapshot.exists) {
print("user:::" + snapshot.value.toString());
final jsonResponse = json.encode(snapshot.value);
List<dynamic> list = json.decode(jsonResponse);
List<Users> users = [];
list.forEach((element) {
Users usersModel = Users.fromJson(element);
users.add(usersModel);
});
users.forEach((element) {
debugPrint("UseX:::" + element.userName);
});
} else {
print('No data available.1');
}
AppUtil.dismissLoader(context: context);
// }).onError((error, stackTrace) {
// print('No data available.2');
// AppUtil.dismissLoader(context: context);
});
}
I just want to update my specific object suppose the users object with the userId=u1 only!
Can somebody help me getting this, update using the query! or I'm doing the wrong way!
Actually I just simple want to update the object based on some condition:
Update Users -> where userId=u1
Thanks in Advance!
Welcome to Flutter Firebase!
First, you should probably change your data structure. There's no need to have a random incremental ID for users. You can use their unique ID as the key for the object:
{
"posts": {}
...
"users": {
"u1": {
"userName": "bla bla 1"
},
"u2": {
"userName": "bla bla 2"
}
}
}
Then, you can do this:
final String userId = 'u1';
final userDoc = FirebaseDatabase.instance.reference().child('users/$userId');
If you're planning on storing a lot of data in posts or users, I highly recommend that you check Cloud Firestore. It offers better structure and more advanced data types.
You're starting with a very flimsy foundation to your database if you go down the route you're taking. A proper schema would look like this:
{
"posts": {
"$postID": {
"postName": "Some Name"
// ...
}
}
"users": {
"$userID": {
"userName": "Some Username"
}
}
}
You would be wise to avoid arrays entirely when using firebase database. This isn't a firebase standard, it's a NoSQL practice.
In short, NoSQL databases use key value pairs to structure data. Since you won't have any two users with the same ids, nor two posts with the same ids, using key-value pairs in your database let you easily accomplish what you're trying to do by design.
I just want to update my specific object suppose the users object with
the userId=u1 only! Can somebody help me getting this, update using
the query! or I'm doing the wrong way!
The way to do this in the user nodes is as simple as this:
String userID = "$userID"; // where $userID is your targeted user
String newUserName = "$newUsername"; // this is the new data you're trying to change
DatabaseReference userRef = FirebaseDatabase.instance.ref().child('users').child(userID);
userRef.update({
"userName": newUserName,
"timestamp": ServerValue.timestamp,
});
I have successfully performed a query on my usersnode object to get the specific users:
Obviously your old implementation won't work when you update your schema to be key-value paired. But that's a good thing because now you can do proper queries which you can then convert into arrays in the app. You would effectively parse through the DatabaseSnapshot value, which is a Map where the keys are the userIDS and the values are corresponding data.
The way you would load the users into your app with is as follows:
DatabaseReference usersRef = FirebaseDatabase.instance.ref().child('users');
usersRef.orderByChild("userName").once(DatabaseEventType.value).then((DatabaseEvent databaseEvent) {
// (You should update to the latest package which has breaking changes like this you need to adjust to)
// First get the snapshot from the new DatabaseEvent object
DataSnapshot snapshot = databaseEvent.snapshot;
// Convert the value into a map you can parse through
Object? snapshotValue = snapshot.value;
Map<dynamic, dynamic> valueMap = snapshotValue is Map<dynamic, dynamic> ? snapshotValue : {};
// Print the results for debugging
if(valueMap.isEmpty) {
debugPrint("No users found");
} else {
debugPrint("${valueMap.length} user${valueMap.length == 1 ? '' : 's'} found");
}
// Loop through the keys of this map (which are the userIDS)
List<Users> users = [];
for(String userID in valueMap.keys) {
Users userModel = Users.fromJson(valueMap[userID]);
users.add(usersModel);
debugPrint("UseX:::" + userModel.userName);
}
AppUtil.dismissLoader(context: context);
});
Lastly, if you allow users to create posts, you might want to consider adding the following node to your schema in addition to just "posts":
"user_posts" {
"$userID": {
"$postID": POST_OBJECT,
// ...
}
}
Then when you want to load the posts created by the user, rather than only being able to query the "posts" node for posts created by the user (which you'll have a limited querying ability), you query the posts under this specific user's posts (by timestamp for example).
If you do it this way, you'd store a copy of the same object under the user's posts node and posts' node.
You seem new to firebase and have a limited understanding of how the database works. It can get very messy very fast and the most important thing to do is understand how you're going to use the data, how it's going to be queried, and most importantly, how you want to be able to sort it (by alphabetical order, by ranking, by number of likes, by date created, etc.).
Understanding these concepts now is very important because it's directly tied to your database security rules as well (which you wouldn't be able to protect with the way you were doing it before).
My advice would be to look up a quick start guide on NoSQL databases (maybe just the firebase documentation starting guide), look at the firebase database security rules guide, understand what fanning out data is, and then start looking into cloud functions so you can do useful things when data is created, updated, or deleted.
It's very easy to neglect this stuff early on while only focusing only on the app, but this is equally important despite not being something your end user would spend a second thinking about. Without a solid database, you'll run into endless issues in the future.
Good luck :)
I am attempting to create a leader board using dynamo db for an quiz style Alexa skill. I have set up the table and users are added to the table with their appropriate data e.g.:
Item: {
"PlatformId": 2,
"UserId": 12345,
"Score": 100,
"NickName": "scott",
"Sport": "football",
}
In my table the Primary key is their UserId, the sort key is the PlatformId (this is the same for all users). I have a secondary global index which sets the platformId as the primary key, and the score as the sort key.
In this leader board i want users to be ranked, the highest scorer being number 1, my first attempt at this was to scan the table using the secondary index, this nicely returned all the users sorted by score, however with the potential to have thousands of users on this leader board, i discovered that the time to scan a table with 10000+ users exceeds the 8 second response time that Alexa skills have. This causes the skill to error and close.
Before the response time exceeded, i was using the LastEvaluatedKey to perform an extra scan if the first one didn't cover the entire table, but on this second scan is when the response time limit was exceeded. Annoyingly it's just taking too long to scan the table.
dbHelper.prototype.scanGetUsers = (ad, newParams = null) => {
return new Promise((resolve, reject) => {
let params = {};
if (newParams != null) {
params = newParams
} else {
params = {
TableName: tableName,
IndexName: 'PlatformId-Score-index',
FilterExpression: "Score >= :s AND PlatformId = :p",
ProjectionExpression: `NickName, Sport, Score`,
// Limit: 10,
ExpressionAttributeValues: {
":p": User.PlatformId,
":s": User.Score,
},
}
}
docClient.scan(params, function (err, data) {
if (err || !data) {
console.error("Unable to read item. Error JSON:", JSON.stringify(err, null, 2));
return reject(JSON.stringify(err, null, 2))
} else {
console.log("scan users data succeeded:", JSON.stringify(data, null, 2));
if(data.LastEvaluatedKey) {
console.log("found a LastEvalutedKey, Continuing scan");
params.ExclusiveStartKey = data.LastEvaluatedKey;
data = data.concat(this.scanGetUsers(ad, params));
}
resolve(data);
}
});
});
}
Is there a way to work around these issues that i haven't explored yet? Or a way to create a leader board with dynamo db that can be structured in an easier way?
You can try Sort Key:
When you combine a partition key and sort key, they create a composite key, and that composite key is the primary key for individual items in a table. With a composite key, you gain the ability to use queries with a KeyConditionExpression against the sort key. In a query, you can use KeyConditionExpression to write conditional statements by using comparison operators that evaluate against a key and limit the items returned. In other words, you can use special operators to include, exclude, and match items by their sort key values.
The article contains all information how to setup and use it.
I have a DynamoDB table with a primary key (_id) being a simple int. I want to get the highest value for the primary key.
How do I return the item in the table with the highest _id?
I can use either the Amazon javascript API or the Dynamoose library.
Partition keys are not stored in order. You would need to scan the entire table, stream over the items, map to the _id attribute and then return the maximum value.
You can easily create Global Secondary Index where _id must to be a sort key and based on it you can make a request like this:
var params = {
TableName: 'Devices',
KeyConditionExpression: 'status = :status',
ScanIndexForward: false, // true = ascending, false = descending
ExpressionAttributeValues: {
':s': status
}
};
docClient.query(params, function(err, data) {});
I have a dynamodb table that stores users videos.
It's structured like this:
{
"userid": 324234234234234234, // Hash key
"videoid": 298374982364723648 // Range key
"user": {
"username": "mario"
}
}
I want to update username for all videos of a specific user. It's possible with a simple update or i have to scan the complete table and update one item a time?
var params = {
TableName: DDB_TABLE_SCENE,
Key: {
userid: userid,
},
UpdateExpression: "SET username = :username",
ExpressionAttributeValues: { ":username": username },
ReturnValues: "ALL_NEW",
ConditionExpression: 'attribute_exists (userid)'
};
docClient.update(params, function(err, data) {
if (err) fn(err, null);
else fn(err, data.Attributes.username);
});
I receive the following error, I suppose the range key is necessary.
ValidationException: The provided key element does not match the schema
Dynamo does not support write operations across multiple items (ie. for more than one item at a time). You will have to first scan/query the table, or otherwise generate a list of all items you'd like to update, and then update them one by one.
Dynamo does provide a batching API but that is still just a way to group updates together in batches of 25 at a time. It's not a proxy for a multi-item update like you're trying to achieve.
I'm dealing with three tables.
Right now I can get the feed of a user (activities table) and activityTypes object (id,type from activitytypes table), but I also need to get related_user (users table) object of a given activity, so I'll know the details of the related_user if the column isn't empty.
users
id
email
username
password_hash
activitytypes
id
type
activities
id
user_id (foreign key from users, owner of the activity)
status
activitytype_id (foreign key from activitytypes)
related_user (foreign key from users.
Shows relation with the user_id, if there's any.
if activitytype_id is 2 or 3, this will be shown as "followed, "commented", etc.)
My models
var activityType = db.Model.extend({
tableName: 'activitytypes',
hasTimestamps: false
});
var activity = db.Model.extend({
tableName: 'activities',
hasTimestamps: true,
activityTypes: function() {
return this.belongsTo(activityType);
}
});
var user = db.Model.extend({
tableName: 'users',
hasTimestamps: true,
feed: function() {
return this.hasMany(activity);
}
});
This is my existing query, how can I add related_user object to it?
user.where({id : 43}).fetchAll({withRelated : ['feed.activityTypes']}).then(function(data) {
data = data.toJSON();
res.send(data);
});
I solved it.
I added a new method to activity, like below.
userRelated : function() {
return this.belongsTo(user,'related_user');
}
And here's the updated query. I don't know whether it's the right way or not in terms of optimization, but it works.
user.where({
id: 43
}).fetchAll({
withRelated: ['feed.userRelated', 'feed.activityTypes']
}).then(function(data) {
data = data.toJSON();
res.send(data);
});
Right now feed, userRelated and activityTypes return without a problem, everything is in data in JSON format.