I have a table storedgames, which contains 2092 items.
And it also has an index on that table, which also lists 2092 items.
when I fetch data, I use the index, to obtain the items for one specific user.
const params = {
TableName: "storedgames",
IndexName: "user-index",
KeyConditionExpression: "#usr = :usr",
ExpressionAttributeNames: { "#usr": "user" },
ExpressionAttributeValues: { ":usr": user }
};
const data = await new Promise((resolve, reject) => {
docClient.query(params, (err, data) => {
if (err) { reject(err); } else { resolve(data); }
});
}).catch((err) => {
console.error(err);
return false;
});
However, the above code does not return all items. It only finds 42. And for today's items there is only 1 hit. When I check directly on the AWS webpage, I actually find more items for today.
And even when I do this using the index, it finds more records.
When I leave out the filtering of the day, I actually find over 130 items,
while my javascript code only returns 42 items when I leave out the day filter.
So my question is, why does the data of my index seem to be incomplete when I call it programmatically ?
The records actually contain a lot of data, and there appears to be a limit in the amount of data that can be fetched per query.
A single Query operation can retrieve a maximum of 1 MB of data. This
limit applies before any FilterExpression is applied to the results.
If LastEvaluatedKey is present in the response and is non-null, you
must paginate the result set (see Paginating the Results).
So, I one possible solution, is to perform multiple fetches until you have the entire collection.
const queryAllItems = (params, callback) => {
let fullResult = { Items: [], Count: 0, ScannedCount: 0 };
const queryExecute = (callback) => {
docClient.query(params, (err, result) => {
if (err) {
callback(err);
return;
}
const { Items, LastEvaluatedKey, Count, ScannedCount } = result;
fullResult.Items = [...fullResult.Items, ...Items];
fullResult.Count += Count;
fullResult.ScannedCount += ScannedCount;
if (!LastEvaluatedKey) {
callback(null, fullResult);
return;
}
params.ExclusiveStartKey = LastEvaluatedKey;
queryExecute(callback);
});
}
queryExecute(callback);
}
Unfortunately, this isn't a complete solution. In my situation, a query for a mere 130 items (which require 4 actual fetches) takes about 15 seconds.
Related
I have a backend method called LikeExists() to verify if a certain user has liked a certain post.
public async Task<bool> LikeExists(int postId)
{
var post = await _postRepository.GetPostByIdAsync(postId);
var user = await _userRepository.GetUserByUsernameAsync(User.GetUsername());
if (_context.Likes.Where(i => i.PostId == post.Id && i.UserId == user.Id).FirstOrDefault() != null) return true;
return false;
}
The method works fine in Postman, but it does not do the job in Angular. If a user presses a like button I first want to check if this user has already liked this post. If he has, he will unlike it and the like will be deleted from the database. If he hasn't liked it, he will like it and the like will be saved in the database.
likeExists(){
this.postService.likeExists(this.post.id).subscribe((response: boolean) =>{
this.like = response;
});
}
likePost() {
if(this.likeExists){
this.postService.likePost(this.post.id, this.model).subscribe((response: Like) => {
this.likee = response;
console.log(response);
this.toastr.success('Liked');
}, error => {
console.log(error);
this.toastr.error(error.error);
});
} else {
this.postService.deleteLike(this.post.id).subscribe(() => {
this.toastr.success('Unliked');
}, error => {
console.log(error);
})
}
}
The problem is it always enters the if{} clause and never the else{} clause. The method below returns an Observable. I think the problem is that it must return a boolean. How can I make this work?
This is the method in the postService:
likeExists(postId: number) {
return this.http.get(this.baseUrl + 'like/exists/' + postId);
}
Try avoiding nested subscription since it will result in an unreadable and hard to maintain code, use rxjs pipes with operators instead, try something like this:
let id= this.post.id;
let likeExists$ = this.postService.likeExists(id);
likeExists$
.pipe(
switchMap(likeExists => {
if (likeExists) {
// delete like
return this.postService.deleteLike(id);
}
// otherwise addlike
return this.postService.addLike(id);
})
).subscribe(
res=> this.toastr.success('Success'),
err=> this.toastr.error('Failed')
);
or even shorter
let id= this.post.id;
let likeExists$ = this.postService.likeExists(id);
likeExists$
.pipe(switchMap(
liked => liked ? this.postService.deleteLike(id) : this.postService.addLike(id)}))
.subscribe(
res=> this.toastr.success('Success'),
err=> this.toastr.error('Failed')
);
The problem is that this.http.get is asynchronous, which means that likeExists returns before this.like is being set. You need to wait for the value to be returned in your observable. Refactor your code to something along these lines:
likePost() {
// Check to see if like exists and wait for response from server
this.postService.likeExists(this.post.id).subscribe((response: boolean) => {
this.like = response;
if (this.like) {
this.postService.likePost(this.post.id, this.model).subscribe((response: Like) => {
this.likee = response;
console.log(response);
this.toastr.success('Liked');
}, error => {
console.log(error);
this.toastr.error(error.error);
});
} else {
this.postService.deleteLike(this.post.id).subscribe(() => {
this.toastr.success('Unliked');
}, error => {
console.log(error);
})
}
});
}
Also, this is a nice guide to asynchronous concepts in general. And the RxJS docs have a bunch of helpful information to get started.
I've implemented a search function which will search and return results from my redux store.
However the result get returned and when I clear the text field the initial state should be return. I end up getting an empty store when I've cleared or entered the wrong keywords.
here is my search & filter function
filter: (state, { payload }) => {
const itemsToFilter = state.filter((item) => {
let itemLowerCase = item.item_description.toLowerCase();
let searchItemToLowerCase = payload.item_description.toLowerCase();
return itemLowerCase.indexOf(searchItemToLowerCase) > -1;
});
if (itemsToFilter) {
return itemsToFilter;
}
return itemInitialState; //initial state of items is not returned
}
and I used useDispatch from react-redux
I'm assuming that in the case where 'you clear the text field' that the result is an event where payload.item_description is ''. In that case, Array.filter() won't match anything, but it will still return an empty array (not null). So, your check of itemsToFilter returns true and your new state is empty. Just check the length of the returned array instead.
filter: (state, { payload }) => {
const itemsToFilter = state.filter((item) => {
...
});
if (itemsToFilter.length === 0) {
return itemsToFilter;
}
return itemInitialState;
}
My DynamoDB table alexas has this item with key "abc" as seen in the DynamoDB console below:
However, the following query returns no result:
const params = { TableName: "alexas",
KeyConditionExpression: "deviceId = :deviceId",
ExpressionAttributeValues: { ":deviceId": "abc"}
}
const docClient = new AWS.DynamoDB.DocumentClient();
docClient.query(params, (err, data) => {
if (err) {
console.error(err);
} else {
console.log(data);
}
});
The above code returns null for err and in data:
{ Items: [], Count: 0, ScannedCount: 0 }
I am new to the DynamoDB style of expressions. Is there anything wrong with my code which I took from here.
If instead of query, I used the scan method and just have TableName in params, I get the items in my table. This confirms that I am performing the operations on the correct table that has data.
The query returned no data because the key value does not match.
The item's deviceId is the string "abc" and not abc. Note the extra quotation marks.
The item was inserted using the DynamoDB console's Create editor and there is no need to include "" if the value is already expected to be of type string.
DynamoDB's Scan operation doesn't take a KeyConditionExpression - only the Query operation takes this parameter. Scan always scans the entire table, and has a FilterExpression to post-filter these results (however please note that you still pay for scanning the entire table).
For example, here is the official documentation of Scan: https://docs.aws.amazon.com/amazondynamodb/latest/APIReference/API_Scan.html
Check QueryAPI
const params = { TableName: "alexas",
KeyConditionExpression: "deviceId = :deviceId",
ExpressionAttributeValues: {
":devideId":{
S: "abc", // here
}
}
}
const docClient = new AWS.DynamoDB.DocumentClient();
docClient.query(params, (err, data) => {
if (err) {
console.error(err);
} else {
console.log(data);
}
});
ExpressionAttributeValues needs to be passed in a different manner.
Update:
Try using Exp attribute names, (I'm not sure if this will make a difference)
var params = {
TableName: "alexas",
KeyConditionExpression: "#d = :dId",
ExpressionAttributeNames:{
"#d": "email"
},
ExpressionAttributeValues: {
":dId": "abc"
}
};
I try to figure out how I can wait for several promises to be executed before the function ends.
Essentially, this is what I try to do:
if a user deletes his/her account, I want to clean up all data which is associated with him
direct data can be deleted
if a user is part of a group, the group shall still exists, if other users are in that group; otherwise the group shall be deleted as well
Here is what I tried so far:
A) Main function (starts the first level of promises):
export function cleanUpAllData(user) {
const userId = user.uid;
const promises = [];
promises.push(deleteCategoriesData(userId)); // this works fine
promises.push(deleteUserAndGroupData(userId)); // this one has other promises which still run when Promise.all() is finished
Promise.all(promises)
.then(() => {
return "ok"; // this works so far, but not all promises are resolved
})
.catch(errPromises => {
console.log("an error occured during the processing of main promises");
console.log(errPromises, errPromises.code);
return "error";
})
}
B) deleteUserAndGroupData function (the other promise is working fine): each group found in the user data starts another level of promises and also triggers a thrid level of promises (deleteGroup) - the rest is working fine
function deleteUserAndGroupData(userId) {
const promisesUserData = [];
return admin.firestore().collection('users').doc(userId).collection('groups').get()
.then(userGroups => {
userGroups.forEach(userGroupData => {
// delete group data
promisesUserData.push(deleteGroups(userId, userGroupData.id)); // here are other promises that need to be resolved - somewhere is a problem
// delete all Groups in Users (subcollection)
promisesUserData.push(deleteGroupInUser(userId, userGroupData.id)); // this works fine
});
Promise.all(promisesUserData)
.then(() => {
admin.firestore().collection('users').doc(userId).delete()
.then(() => {
return "user data deleted"; // works fine
})
.catch(() => {
console.log("an error occured during deleting of user");
return "error";
});
})
.catch(errPromises => {
console.log("an error occured during the processing of promisesUserData");
console.log(errPromises, errPromises.code);
return "error";
})
})
.catch(errUserGroups => {
console.log(errUserGroups, errUserGroups.code);
return "no groups in User";
});
}
C) deleteGroups functions: deletes the sub collections in the group (works fine) and after that the group shall be deleted if there is no other user (which does not work)
function deleteGroups(userId,groupId) {
const promisesDeleteGroups = [];
// delete groups subcollection data
promisesDeleteGroups.push(deleteGroupRole(userId, groupId));
promisesDeleteGroups.push(deleteGroupUser(userId, groupId));
return Promise.all(promisesDeleteGroups).then(() => {
checkForOthers(groupId);
}).catch(errDeleteGroupSubcollections => {
console.log("an error occured during the processing of promisesDeleteGroups");
console.log(errDeleteGroupSubcollections, errDeleteGroupSubcollections.code);
return "error";
});
}
D) checkForOthers function - checks if there is any entry in the subcollection and shall start to delete the group (but does not)
function checkForOthers(groupId) {
return admin.firestore().collection('groups').doc(groupId).collection('users').get()
.then(groupUsers => {
return "other users exist - group should not be deleted";
})
.catch(errGroupUsers => {
// console.log("no other group members - group can be deleted");
// return "no other group members - group can be deleted";
console.log(errGroupUsers, errGroupUsers.code);
checkForInvitesAndDelete(groupId);
});;
}
E) checkForInvitesAndDelete: first I want to delete another subcollection which might or might not exists, if another user has been invited to the group; then it should trigger the final group delete (which seems not to work)
function checkForInvitesAndDelete(groupId) {
const promisesInvitesAndDelete = [];
return admin.firestore().collection('groups').doc(groupId).collection('invitations').get()
.then(groupInvitations => {
console.log("delete open Group Invitations");
groupInvitations.forEach(groupInvite => {
promisesInvitesAndDelete.push(deleteGroupInvite(groupId, groupInvite.id));
});
Promise.all(promisesInvitesAndDelete)
.then(() => {
deleteGroup(groupId)
})
.catch(errPromisesInvitesAndDelete => {
console.log("an error occured during the processing of deleting group invites");
console.log(errPromisesInvitesAndDelete, errPromisesInvitesAndDelete.code);
return "error";
});
})
.catch(() => {
console.log("no open invitations");
deleteGroup(groupId);
});
}
F) deleteGroup function
function deleteGroup(groupId) {
return admin.firestore().collection('groups').doc(groupId).delete();
}
I am relatively new to programming, especially Firebase Functions, so any help would be appreciated!!
Thank you!
You are not using the return keyword everywhere, where it should be. If you do a async task, you must 'return' it someway.
Some examples:
example A: add return before Promise.all(promises)
... B: add return before Promise.all(promisesUserData)
... C: add return before checkForOthers(groupId)
... D: add return before checkForInvitesAndDelete(groupId)
... E: add return before Promise.all(promisesInvitesAndDelete) and deleteGroup(groupId)
I added the 'return' statements which helped a lot, but that was not the full answer.
In (D) I thought that if my collection has no data, it would run into the 'catch' phase, but it is not. So, I needed to check for an empty result set in my 'then' phase.
if (groupUsers.empty) {
return checkForInvitesAndDelete(groupId);
} else {
return "other users exist - group should not be deleted";
}
Same with the function (E) when there is no open invitation.
I am trying to learn more about the Http.Get. I am using a json file to mock the HTTP call. After the call I want to limit the rows to unique states. I know there is an Rxjs command called distinct (help on Rxjs distinct). However I do not understand the syntax for the distinct. When I run this code as is I get an array of states. However when I add the distinct it still has duplicated states like Texas.
public getStates(): Observable<IState[]> {
return this._http.get(this.stateUrl)
.map((res: Response) => <IState[]>res.json())
// distinct by state name
.distinct((x) => return x.state)
;
}
Here is the interface for IState
export interface IState {
id: number;
state: string;
city: string;
name: string;
}
Trying to only get rows with a unique state.
I have repo for the code on this Github project
You're confusing a bit the meaning of the distinct in this situation!
This distinct is meant to filter the response from your Observable, not its content! Here you're trying to "distinct" (filter) your json object, but the distinct will filter the responses (multiple) "resolved" by the Observable. So, if the observable "resolves" the same response multiple times, you'll receive only once, because of your .distinct.
This means, if the 1st time the Observable returns you back one json (e.g [{0: a, 1:b, 2:c}]) and the second time he returns the same json reponse, then your "subscribers" will not receive it twice, because the .distinct will "filter" it and as a result it will not be fired!
more details here: https://github.com/Reactive-Extensions/RxJS/blob/master/doc/api/core/operators/distinct.md
So, what you're looking for, is how to "filter" duplicated results within your json.
The answer is simple: use a loop to check its properties/values or some library like underscore which might have this method already implemented.
The IState is an object. You have to define what exactly makes them distinct from each other.You can look at the documentation, there is a way to define what makes an object distinct
For example:
/* Without key selector */
var source = Rx.Observable.of(42, 24, 42, 24)
.distinct();
var subscription = source.subscribe(
function (x) {
console.log('Next: %s', x);
},
function (err) {
console.log('Error: %s', err);
},
function () {
console.log('Completed');
});
// => Next: 42
// => Next: 24
// => Completed
/* With key selector */
var source = Rx.Observable.of({value: 42}, {value: 24}, {value: 42}, {value: 24})
.distinct(function (x) { return x.value; });
var subscription = source.subscribe(
function (x) {
console.log('Next: %s', x);
},
function (err) {
console.log('Error: %s', err);
},
function () {
console.log('Completed');
});
// => Next: { value: 42 }
// => Next: { value: 24 }
// => Completed
The problem is that when you're calling .distinct((x) => return x.state) the x variable is the entire array from https://github.com/ATXGearHead12/distinct/blob/master/src/api/state.json. Then property x.state doesn't exists.
So you need to call distinct on each item in the array not on the array itself.
For example like the following:
this._http.get(this.stateUrl)
.map((res: Response) => <IState[]>res.json())
.concatMap(arr =>
Observable.from(arr).distinct((x) => return x.state)
)
.toArray(); // collect all items into an array