I have a DynamoDB database with an attribute Event_Name which has uppercase values, for example KRISHNA. When I specify a Scan FilterExpression comparitor CONTAINS with a lowercase value, for example krishna, the item with value KRISHNA is not returned. When I use the uppercase value it returns the item. Please help me.
For reference my code is:
var params = {
TableName: "User",
FilterExpression: "NOT userId in (:a) and contains(Event_Name, :name)",
ExpressionAttributeValues: {
":a": {
S: $scope.userid
},
":name": {
S: namekey
}
}
};
using dynamodb scan method
Probably you already figured out, but since I stumbled upon this question and it's not closed, here is a link in AWS forum addressing the issue
https://forums.aws.amazon.com/thread.jspa?threadID=92159
DynamoDB is case sensitive. If your data is case insensitive, one solution is to lower case or upper case the data before storing it in DynamoDB. Then you can get around this by querying for all lower case or all upper case. You will need to take locale into account for locale-sensitive ordering.
So there is nothing wrong you are doing, you just were expecting something that is not available with DynamoDB
Related
First question on StackOverflow, long time reader first time poster or whatever people say.
I'm developing a Discord bot in my free time using Discord.js, and I'm using Sequelize to interface with a local SQLite database. I can insert data into it just fine-- however, I can't seem to delete any of the records I add. Relevant piece of code is below, which I believe to be self-contradictory:
const query3 = await Towers.findAll({
attributes: ['channelID']
});
console.log(JSON.stringify(query3)); //returns the one Tower
console.log(query3[0].channelID === channel); //returns true(!)
const query2 = await Towers.findAll({
attributes: ['channelID'],
where: {channelID: channel}
});
console.log(JSON.stringify(query2)); //returns empty
//DELETE FROM Towers WHERE channelID = channel;
const query = await Towers.destroy({
where: {channelID: channel}
});
console.log(query); //returns 0, expected behavior given query2 returns empty
I'm attempting to delete a record from a table named Towers by passing a channel ID to it, which is expected to be unique. However, when I make any query on the database with a WHERE clause, the query returns an empty set-- even when, in this example, I sanity-checked and verified that the value I'm attempting to remove is present in the table. This occurs for both findAll() and findOne() as long as a WHERE clause is present.
(For posterity, I've double and triple checked that channelID was spelled correctly and with the correct capitalization in all instances.)
I'm happy to provide any more information if needed!
EDIT: As requested, the model definition...
const Towers = sequelize.define('Towers', {
serverID: {
type: Sequelize.INTEGER,
allowNull: false,
},
channelID: {
type: Sequelize.INTEGER,
unique: true,
allowNull: false,
},
pattern: Sequelize.STRING,
height: Sequelize.INTEGER,
delay: Sequelize.BOOLEAN,
});
channel in the snippet in the original post is defined as parseInt(interaction.options.getChannel('channel').id).
To anyone who happens to have the same issue I did, the answer is a doozy.
I wanted to store Discord server and channel ID's as integers, even though they're returned to you as strings when calling the API. As it turns out, Discord snowflakes are higher than float64 precision, which JS uses. When parsing the strings into integers to insert them into my table, the value changed from the intended number, and I was creating erroneous records.
In my case (with the actual numbers obfuscated) interaction.options.getChannel('channel').id returned "837512533934092340", while parseInt(interaction.options.getChannel('channel').id returned 837512533934092300. The number I was adding to the table was somehow 40 less!
I'm not sure if this could be fixed by using BigInt, but since it's going into a different structure anyway, I just shrugged and changed the serverId and channelId types to Sequelize.STRING in the model definition and removed the parseInt calls. Works like a charm now.
Good opportunity to shake my fist at JS though.
I am working on a 'typeahead’ type function which will check my Database with the current typed text to provide search suggestions of users using Felgo.
Here is the link for Felgos Firebase documentation
As to not search every entry I am looking to use the startAt and limitTo for a lower data use.
However when applying the startAt my searches only return undefined, I have tried testing this by changing my startAt from a variable to explicit data but this still only returns undefined.
My function is below:
function searchUsers(searchString) {
db.getValue("public/nameList/", {
orderByChild: true,
startAt: searchString, //searchString is a variable with my .currentText to search.
limitToFirst: 10,
}, function(success, key, value) {
if(success) {
searchArr = []
searchArr = value
console.debug("Read user value for key", key, "from DB:", value)
}
})
}
I have also tried by passing my var searchString through JSON.stringify(searchString) and also return undefined!
Removing the startAt: query entirely returns the entire result of nameList as expected, but no matter how I try to implement my startAt it always returns undefined.
A sample of my nameList JSON is:
nameList: {
"EddieLaw245" : 530343772383,
"EddieLawrence91" : 530343772385,
"EdwardL91" : 530343772386,
"EdwardLaw" : 530343772384,
"Edwardlawrence91" : 530343772380,
"JoBrownLondon" : 530343772381,
"KatiePrescottHair" : 543592635596,
"Tracey-Sweeting" : 530343772382
}
So with the above example, When I type E it should remove the last 3 entries, and so on.
The problem is that you're specifying orderByChild: true. If we look at the documentation of that:
orderByChild: If present, the queried object will have its properties ordered by values at sub-paths defined by the value of this property. Ordering by child properties makes the filter properties startAt, endAt and equalTo filter by the child property values
It may not be immediately clear from this, but orderByChild allows you to order the results on a property value under each of those nodes. So your code tries to order the child nodes on the value of a property true, which isn't possible (and should actually generate a compile-time error in the library) as the nodes under nameList don't have any child properties of their own. They merely have a key and a value.
What you're looking for is orderByKeys, which orders the child nodes on their keys. So:
db.getValue("public/nameList/", {
orderByKeys: true,
startAt: searchString,
limitToFirst: 10,
}
You'll typically also want to specify an endAt value, to ensure your type-ahead only shows values that start with the search string. If you only allow ASCII values in the keys, the simplest way to do this is:
startAt: searchString,
endAt: searchString + "~",
The ~ here is no magic operator, but merely the last ASCII characters. If you want to allow a broader character set, you'll need to use the last character in that character set - for example \uF7FF is the last code point for Unicode.
Update from OP
Though I'm certian Franks correct with typical Firebase usage; I suspect due to the Felgo plugin I am using the full solution has a slight adjustment;
db.getValue("public/nameList/", {
"orderByKey": true,
"startAt": searchString,
"endAt": searchString+"~",
"limitToFirst": 10,
}, function(success, key, value) {....}
})
Notes on the above - my filters/queries are surrounded by quotation marks "startAt", also instead of orderByKeys, I have used orderByKey
I have a DynamoDB table, with a composite key, which looks like this:
PK
SK
Type
Email
Description
USER#A
USER#A
User
a#example.com
USER#A
BUG#1
Bug
This looks ok
USER#B
BUG#2
Bug
My user wasn't created first!
I'd like to ensure that a "User" record exists before adding a related "Bug" record - So the 3rd item here is incorrect.
When I put a bug item with the condition attribute_exists(PK), the condition is never true. When I remove the condition, I end up with a that third row; A Bug with no corresponding User.
My understanding is that attribute_exists() only looks at items with the combined composite key, and not across the whole table, regardless of which attribute you supply.
Is there a method of ensuring an item with the same Partition Key exists, while ignoring the Sort Key in this scenario?
DynamoDB condition expressions can be confusing, and the docs can compound that problem!
The DynamoDB condition expression works by 1) finding the item, 2) evaluating the condition expression, and finally 3) writing to the database if the condition evaluates to true.
I assume your put operation looks something like this:
ddbClient.put({
TableName: "YOUR TABLE",
Item: {
PK: "USER#B",
SK: "BUG#2",
Type "Bug",
Description: "My user wasn't created first!"
},
ConditionExpression: "attribute_exists(PK)"
})
In this example, DynamoDB first tries to find the item with PK: "USER#B" SK: "BUG#2", which does not exist. As you're experiencing, this item will not be written to DynamoDB because an item with that primary key does not exist.
The problem you are seeing, as you've alluded to in your question, is that a CondttionExpression applies to only a single item. However, you are trying to conditionally put an item in the database by applying the condition to another item. That is a great candidate for a DynamoDB transaction.
Transactions let you group operations together in an all-or-nothing operation. If one of the operations in your transaction fails, the entire transaction will fail and none of the operations will apply.
You can achieve what you are after by taking this approach
ddbClient.transactWriteItems({
TransactItems=[
{ "PUT":
{
TableName: "YOUR TABLE",
Item: {
PK: "USER#B",
SK: "BUG#2",
Type "Bug"
}
}
},
{ "ConditionCheck":
{
TableName: "YOUR TABLE",
Item: {
PK: "USER#B",
SK: "USER#B"
},
ConditionExpression: "attribute_exists(PK)"
}
}
]
})
In the above transaction, I'm using a ConditionCheck to confirm the existence of a user before entering the bug. If the user does not exist, the transaction will fail and the bug won't be written to DDB.
For a more thorough explanation of DynamoDB Condition Expressions, I highly recommend you check out Understanding DynamoDB Condition Expressions by Alex Debrie.
I have a NodeJS app using Express-Js framework and the latest Express-Validator package. I have an input field that I want to "escape" and save the result in AWS DynamoDB.
The parameter is req.body.dealer_code and the value...
Before
req.body.dealer_code = "<script>alert('hello')</script>"
After
req.body.dealer_code = "<script>alert('hello')</script>"
When the value is inserted into the DynamoDB, it appears as "<script>alert('hello')</script>" in the Tree view.
What goes wrong here?
Thanks.
After an experiment, the string is indeed stored as an escaped string.
{ Item:
{ escaped: '<script>alert('hello')</script>'
}
}
I am fairly new in this realm and any help is appreciated
I have a table in Dynamodb database named Tenant as below:
"TenantId" is the hash primary key and I have no other keys. And I have a field named "IsDeleted" which is boolean
Table Structure
I am trying to run a query to get the record with specified "TenantId" while it is not deleted ("IsDeleted == 0")
I can get a correct result by running the following code: (returns 0 item)
var filter = new QueryFilter("TenantId", QueryOperator.Equal, "2235ed82-41ec-42b2-bd1c-d94fba2cf9cc");
filter.AddCondition("IsDeleted", QueryOperator.Equal, 0);
var dbTenant = await
_genericRepository.FromQueryAsync(new QueryOperationConfig
{
Filter = filter
}).GetRemainingAsync();
But no luck when I try to get it with following code snippet (It returns the item which is also deleted) (returns 1 item)
var queryFilter = new List<ScanCondition>();
var scanCondition = new ScanCondition("IsDeleted", ScanOperator.Equal, new object[]{0});
queryFilter.Add(scanCondition);
var dbTenant2 = await
_genericRepository.LoadAsync("2235ed82-41ec-42b2-bd1c-d94fba2cf9cc", new DynamoDBOperationConfig
{
QueryFilter = queryFilter,
ConditionalOperator = ConditionalOperatorValues.And
});
Any Idea why ScanCondition has no effect?
Later I also tried this: (throw exception)
var dbTenant2 = await
_genericRepository.QueryAsync("2235ed82-41ec-42b2-bd1c-d94fba2cf9cc", new DynamoDBOperationConfig()
{
QueryFilter = new List<ScanCondition>()
{
new ScanCondition("IsDeleted", ScanOperator.Equal, 0)
}
}).GetRemainingAsync();
It throws with: "Message": "Must have one range key or a GSI index defined for the table Tenants"
Why does it complain about Range key or Index? I'm calling
public AsyncSearch<T> QueryAsync<T>(object hashKeyValue, DynamoDBOperationConfig operationConfig = null);
You simply cant query a table only giving a single primary key (only hash key). Because there is one and only one item for that primary key. The result of the Query would be that still that single item, which is actually Load operation not Query. You can only query if you have composite primary key in this case (Hash (TenantID) and Range Key) or GSI (which doesn't impose key uniqueness therefore accepts duplicate keys on index).
The second code attempts to filter the Load. DynamoDBOperationConfig's QueryFilter has a description ...
// Summary:
// Query filter for the Query operation operation. Evaluates the query results and
// returns only the matching values. If you specify more than one condition, then
// by default all of the conditions must evaluate to true. To match only some conditions,
// set ConditionalOperator to Or. Note: Conditions must be against non-key properties.
So works only with Query operations
Edit: So after reading your comments on this...
I dont think there conditional expressions are for read operations. AWS documents indicates they are for put or update operations. However, not being entirely sure on this since I never needed to do a conditional Load. There is no such thing like CheckIfExists functionality as well in general. You have to read the item and see if it exists. Conditional load will still consume read throughput so your only advantage would be only NOT retrieving it in other words saving the bandwith (which is very negligible for single item).
My suggestion is read it and filter it in your application layer. Dont query for it. However what you can also do is if you very need it you can use TenantId as hashkey and isDeleted for range key. If you do so, you always have to query when you wanna get a tenant. With the query you can set rangeKey(isDeleted) to 0 or 1. This isnt how I would do it. As I said, would just read it and filter it at my application.
Another suggestion thing could be setting a GSI on isDeleted field and writing null when it is 0. This way you can only see that attribute in your table when its only 1. GSI on such attribute is called sparse index. Later if you need to get all the tenants that are deleted (isDeleted=1) you can simply scan that entire index without conditions. When you are writing null when its 0 dynamoDB wont put it in the index at the first place.