I have the following code. I was able to use a variable for the table name, and for the search param. But I don't seem to be able to get tableKey as a variable to work. When I run the code I get "Error Lambda: ValidationException: The provided key element does not match the schema". tableKey is set in this case to "PhoneNumber" which is the name of the field in my DDB table.
Is this possible to use a variable in this location?
async function handleRequest(searchParam,dbTable,tableKey) {
let Details = {
TableName: dbTable,
Key: {
tableKey: searchParam,
}
};
return docClient.get(Details).promise();
}
You cannot pass in the keys as a variable, only the values.
Actually, I was able to figure it out.
async function handleRequest(searchParam, dbTable, tableKey) {
let Details = {
TableName: dbTable,
Key: {
}
};
Details.Key[tableKey] = searchParam;
return docClient.get(Details).promise();
Related
I'm new to DynamoDB and trying to query a table based off the presence of a list of certain values for a field.
I have a field doc_id, which is also a secondary index, and I'd like to return all results where doc_id is contained in a list of values.
I'm trying something like this:
response = table.query(
IndexName='doc_id-index',
FilterExpression=In(['27242226'])
)
But clearly that is not correct.
Can anyone point me in the right direction?
Thanks!
with Query operation
A FilterExpression does not allow key attributes. You cannot define a filter expression based on a partition key or a sort key.
So, Your doc_id field is the partition key of the doc_id-index and cannot be used in FilterExpression.
Note
A FilterExpression is applied after the items have already been read; the process of filtering does not consume any additional read capacity units.
I'm assuming you have another field like userId, just to show how to implement IN operation.(Query)
var params = {
TableName: 'tbl',
IndexName: 'doc_id-index',
KeyConditionExpression: 'doc_id= :doc_id',
FilterExpression: 'userId IN (:userId1,:userId2)',//you can add more userId here
ExpressionAttributeValues: {
':doc_id':100,
':userId1':11,
':userId2':12
}
};
If you have more userId you should pass to FilterExpression dynamically.
but in your case, you can use Scan operation
var params = {
TableName : "tbl",
FilterExpression : "doc_id IN (:doc_id1, :doc_id2)",
ExpressionAttributeValues : {
":doc_id1" :100,
":doc_id2" :101
}
};
and even pass to FilterExpression dynamically like below
var documentsId = ["100", "101","200",...];
var documentsObj = {};
var index = 0;
documentsId.forEach((value)=> {
index++;
var documentKey = ":doc_id"+index;
documentsObj[documentKey.toString()] = value;
});
var params = {
TableName: 'job',
FilterExpression: 'doc_id IN ('+Object.keys(documentsObj).toString()+')',
ExpressionAttributeValues: documentsObj,
};
Note:be careful while using Scan operation, less efficient than Query.
My DynamoDB table alexas has this item with key "abc" as seen in the DynamoDB console below:
However, the following query returns no result:
const params = { TableName: "alexas",
KeyConditionExpression: "deviceId = :deviceId",
ExpressionAttributeValues: { ":deviceId": "abc"}
}
const docClient = new AWS.DynamoDB.DocumentClient();
docClient.query(params, (err, data) => {
if (err) {
console.error(err);
} else {
console.log(data);
}
});
The above code returns null for err and in data:
{ Items: [], Count: 0, ScannedCount: 0 }
I am new to the DynamoDB style of expressions. Is there anything wrong with my code which I took from here.
If instead of query, I used the scan method and just have TableName in params, I get the items in my table. This confirms that I am performing the operations on the correct table that has data.
The query returned no data because the key value does not match.
The item's deviceId is the string "abc" and not abc. Note the extra quotation marks.
The item was inserted using the DynamoDB console's Create editor and there is no need to include "" if the value is already expected to be of type string.
DynamoDB's Scan operation doesn't take a KeyConditionExpression - only the Query operation takes this parameter. Scan always scans the entire table, and has a FilterExpression to post-filter these results (however please note that you still pay for scanning the entire table).
For example, here is the official documentation of Scan: https://docs.aws.amazon.com/amazondynamodb/latest/APIReference/API_Scan.html
Check QueryAPI
const params = { TableName: "alexas",
KeyConditionExpression: "deviceId = :deviceId",
ExpressionAttributeValues: {
":devideId":{
S: "abc", // here
}
}
}
const docClient = new AWS.DynamoDB.DocumentClient();
docClient.query(params, (err, data) => {
if (err) {
console.error(err);
} else {
console.log(data);
}
});
ExpressionAttributeValues needs to be passed in a different manner.
Update:
Try using Exp attribute names, (I'm not sure if this will make a difference)
var params = {
TableName: "alexas",
KeyConditionExpression: "#d = :dId",
ExpressionAttributeNames:{
"#d": "email"
},
ExpressionAttributeValues: {
":dId": "abc"
}
};
I was assuming that it was possible to do something like:
transaction.add(collectionRef,{
uid: userId,
name: name,
fsTimestamp: firebase.firestore.Timestamp.now(),
});
But apparently it is not:
transaction.add is not a function
The above message is displayed inside the chrome console.
I see that we can use the set method of the transaction to add a new document transactionally. see: https://firebase.google.com/docs/firestore/manage-data/transactions
The thing is if I use set instead of add(which is not supported anyways), the id of the document should be created by me manually, firestore won't create it.
see: https://firebase.google.com/docs/firestore/manage-data/add-data
Do you see any downside of this not having an add method that generates the id for you automatically?
For example, is it possible that the id generated by the firestore itself is somehow optimized considering various concerns including performance?
Which library/method do you use to create your document IDs in react-native while using transaction.set?
Thanks
If you want to generate a unique ID for later use in creating a document in a transaction, all you have to do is use CollectionReference.doc() with no parameters to generate a DocumentReference which you can set() later in a transaction.
(What you're proposing in your answer is way more work for the same effect.)
// Create a reference to a document that doesn't exist yet, it has a random id
const newDocRef = db.collection('coll').doc();
// Then, later in a transaction:
transaction.set(newDocRef, { ... });
after some more digging I found in the source code of the firestore itself the below class/method for id generation:
export class AutoId {
static newId(): string {
// Alphanumeric characters
const chars =
'ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789';
let autoId = '';
for (let i = 0; i < 20; i++) {
autoId += chars.charAt(Math.floor(Math.random() * chars.length));
}
assert(autoId.length === 20, 'Invalid auto ID: ' + autoId);
return autoId;
}
}
see: https://github.com/firebase/firebase-js-sdk/blob/73a586c92afe3f39a844b2be86086fddb6877bb7/packages/firestore/src/util/misc.ts#L36
I extracted the method (except the assert statement) and put it inside a method in my code. Then I used the set method of the transaction as below:
generateFirestoreId(){
const chars = 'ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789';
let autoId = '';
for (let i = 0; i < 20; i++) {
autoId += chars.charAt(Math.floor(Math.random() * chars.length));
}
//assert(autoId.length === 20, 'Invalid auto ID: ' + autoId);
return autoId;
}
then,
newDocRef = db.collection("PARENTCOLL").doc(PARENTDOCID).collection('SUBCOLL').doc(this.generateFirestoreId());
transaction.set(newDocRef,{
uid: userId,
name: name,
fsTimestamp: firebase.firestore.Timestamp.now(),
});
Since I am using the same algo for the id generation as the firestore itself I feel better.
Hope this helps/guides someone.
Cheers.
Based on the answer from Doug Stevenson, this is how I got it worked with #angular/fire:
// Create a reference to a document and provide it a random id, e.g. by using uuidv4
const newDocRef = this.db.collection('coll').doc(uuidv4()).ref;
// In the transaction:
transaction.set(newDocRef, { ... });
To complete Stefan's answer. For those using Angularfire, earlier to version 5.2 using CollectionReference.doc() results in an error "CollectionReference.doc() requires its first argument to be of type non-empty string".
This workaround worked for me:
const id = this.afs.createId();
const ref = this.afs.collection(this.collectionRef).doc(id);
transaction.set(ref, { ... });
Credit: https://github.com/angular/angularfire/issues/1974#issuecomment-448449448
I'd like to add an answer solving the id problem. There's no need to generate your own ids. The documentReference is updated after the transaction.set() is called, so in order to access the Firestore's id you need to just do the following:
const docRef = collectionRef.doc();
const result = await transaction.set(docRef, input);
const id = docRef.id;
First of all, firestore transaction object has 4 (get,set,update,delete) methods and doesnt has "add" method. However, the "set" method can be used instead.
import { collection,doc,runTransaction } from "firebase/firestore";
On the other hand documentReference must be created for "set" method.
Steps :
1-) collection method create a collectionReference object.
const collectionRef = collection(FirebaseDb,"[colpath]");
2-) doc method create a documentReference object with unique random id for specified collectionReference.
const documentRef = doc(collectionRef);
3-) add operation can be performed with the transaction set method
try {
await runTransaction(FirebaseDb,async (transaction) => {
await transaction.set(documentRef, {
uid: userId,
name: name,
fsTimestamp: firebase.firestore.Timestamp.now(),
});
})
} catch (e) {
console.error("Error : ", e);
}
My objects in Dynamodb look roughly like this:
{
userId: "GEFOeE8EsaWmq4NQ3oh7tbeVkLx1",
url: 'objectURL',
object: {}
}
I have this simple piece of code for deleting an object, when the user that owns the object requests a delete. The user argument here is a parsed JWT, by the way.
export async function deleteObject(user, url) {
let params = {
TableName: OBJECTS_TABLE,
Key: {
url: url,
},
ConditionExpression: `userId = :uid`,
ExpressionAttributeValues: {
":uid": {
S: user.sub
}
}
};
let deleteResult = await dynamoDb.delete(params).promise();
return deleteResult;
}
The problem is that it doesn't work, and I've made sure that the problem stems from the ConditionExpression by changing = to <>. I simply get this:
ConditionalCheckFailedException: The conditional request failed
I'm sure solving the problem wouldn't be difficult, but I barely have any information
Questions:
Why is the condition expression failing? Everything looks alright, and it should work. Right?
How could I debug this issue better?
The await/async is not supported by AWS SDK at the moment. Please refer this similar issue.
The SDK currently relies on CLS to trace the call context. It doesn't
work with async/await functionality right now. You can see the
discussion here.
It should work if you remove the await. Example below:-
let deleteResult = dynamodb.deleteItem(params).promise();
deleteResult.then(function (data) {
console.error("Delete item result :", JSON.stringify(data,
null, 2));
}).catch(function (err) {
console.error("Delete item result error :", JSON.stringify(err,
null, 2));
});
I figured it out. ExpressionAttributeValues can be directed used, without mentioning the datatype. The Javascript SDK does that automatically.
export async function deleteObject(user, url) {
let params = {
TableName: OBJECTS_TABLE,
Key: {
url: url,
},
ConditionExpression: `userId = :uid`,
ExpressionAttributeValues: {
":uid": user.sub
}
};
let deleteResult = await dynamoDb.delete(params).promise();
return deleteResult;
}
I have the following in my initialize file to get the values loaded in the database on startup:
Meteor.startup(function() {
if(typeof Person.findOne() === 'undefined') {
Person.insert({
name: "",
gender: ["male", "female", "prefer not to say"],
age: 0
});
}
});
And then in the server/abc.js I have:
Meteor.methods({
checkPerson: function (input) {
for (var key in Person) {
if (input === key) {
...
}
}
}
});
This meteor method checkPerson is called in the client side with a string value being passed as its only argument(input).
I want to check this 'input' string value against the name of the key in the Person Collection.
Person has a key called 'gender'. So for instance, if the 'input' holds the string value 'gender' then the if statement should be true but in my case it comes as false and hence the code inside the if statement is never executed.
Any help/guidance with this will be appreciated.
UPDATE
I searched on mongodb documentation and found here: http://docs.mongodb.org/manual/reference/operator/query/exists/ and also using some help from this thread: (using $exists in Mongo with dynamic key names and the native driver for node)
that I could do something like this:
var checkThis = {};
checkThis[input] = { $exists : true };
var p = Person.findOne(checkThis);
So if it finds one then 'p' holds the record or else it will be undefined. But still the above code does not work.
If I were to put directly:
var p = Person.find({gender: {$exists: true} });
then it works.
So I need assistance in getting the code to work with the variable 'input'.
Mongo is a schemaless database - you can insert any document structure you like into a collection and the data store won't complain. Therefore Person won't be able to indicate which fields conform to the pattern.
The most common way people deal with this problem is to use a package which provides a schema layer on top of mongo. With meteor, a popular choice is SimpleSchema, and its related package AutoForm. SimpleSchema allows you to define which fields should be allowed into a collection, and AutoForm gives you a set of helpers to enforce them in your UI.
If, instead, you prefer not to use a package you could do something like the following:
person.js
var REQUIRED_FIELDS = {
name: String,
gender: ['male', 'female', 'prefer not to say'],
age: Number
};
Person = new Meteor.Collection('person');
Person.isValid = function(person) {
try {
check(person, REQUIRED_FIELDS);
return true;
} catch (_error) {
return false;
}
};
Meteor.methods({
'person.insert': function(person) {
check(person, REQUIRED_FIELDS);
return Person.insert(person);
}
});
my-template.js
Template.myTemplate.events({
submit: function() {
var person = {
name: $('#name').val(),
gender: $('#gender').val(),
age: parseInt($('#age').val(), 10)
};
if (Person.isValid(person))
Meteor.call('person.insert', person);
else
alert('invalid person');
}
});
Here we are using meteor's check package to do some basic field validation. By adding an isValid helper to the Person collection, we can validate the schema without the need for a method call. Best of all we can reuse the same check when inserting a new document.