I am new to DynamoDB and wanted to know how can we query on a table in DynamoDB by using ONLY partition key in JAVA
I have table called "ervive-pdi-data-invalid-qa" and it's Schema is :
partition key is "SubmissionId"
Sort key is "Id".
City (Attribute)
Errors (Attribute)
The table looks like this:
Table
I want to retrieve the sort key value and remaining attributes data by using Partition key using (software.amazon.awssdk) new version of AWS SDK DynamoDB classes.
is it possible to get it? If so, can any one post the answers?
Have tried this:
DynamoDbClient ddb =
DynamoDbClient.builder().region(Region.US_EAST_1).build();
DynamoDbEnhancedClient enhancedClient =
DynamoDbEnhancedClient.builder()
.dynamoDbClient(ddb)
.build();
//Define table
DynamoDbTable<ErvivePdiDataInvalidQa> table =
enhancedClient.table("ervive-pdi-data-invalid-qa",
TableSchema.fromBean(ErvivePdiDataInvalidQa.class));
Key key = Key.builder().partitionValue(2023).build();
ErvivePdiDataInvalidQa result = table.getItem(r->r.key(key));
System.out.println("The record id is "+result.getId());
ErvivePdiDataInvalidQa table class is in below comment*
and it is returning "The provided key element does not match the schema (Service: DynamoDb, Status Code: 400, Request ID: PE1MKPMQ9MLT51OLJQVDCURQGBVV4KQNSO5AEMVJF66Q9ASUAAJG, Extended Request ID: null)"
Query you need is documented in one of the examples of AWS Dynamodb Query API for Java.
AmazonDynamoDB client = AmazonDynamoDBClientBuilder.standard()
.withRegion(Regions.US_WEST_2).build();
DynamoDB dynamoDB = new DynamoDB(client);
Table table = dynamoDB.getTable("ervive-pdi-data-invalid-qa");
QuerySpec spec = new QuerySpec()
.withKeyConditionExpression("SubmissionId = :v_id")
.withValueMap(new ValueMap()
.withInt(":v_id", 2146));
ItemCollection<QueryOutcome> items = table.query(spec);
Iterator<Item> iterator = items.iterator();
Item item = null;
while (iterator.hasNext()) {
item = iterator.next();
System.out.println(item.toJSONPretty());
}
A single Query operation can retrieve a maximum of 1 MB of data, see documentation
I have been working with Padma on this issue. We first tried A. Khan's code but could not get passed authentication with v1. Instead we got "WARNING: Your profile name includes a 'profile ' prefix. This is considered part of the profile name in the Java SDK, so you will need to include this prefix in your profile name when you reference this profile from your Java code."
ultimately it could not get the credentials. Our credentials assume IAM roles in .aws/config-i2 file. It works fine in v2 but not v1.
So then we tried v2 of the SDK and have no problems with connecting but we get NULL returned on trying to fetch all records from the table.
In all of the below attempts using v2 of SDK, table data returns NULL
We created this table class
package data;
import software.amazon.awssdk.enhanced.dynamodb.mapper.annotations.DynamoDbBean;
import software.amazon.awssdk.enhanced.dynamodb.mapper.annotations.DynamoDbPartitionKey;
import software.amazon.awssdk.enhanced.dynamodb.mapper.annotations.DynamoDbSortKey;
#DynamoDbBean
public class ErvivePdiDataInvalidQa {
private int submissionId;
private String id;
private String address1;
private String city;
private String dateOfBirth;
private String errors;
private String firstName;
private String firstNameNormalized;
private String gender;
private String lastName;
private String lastNameNormalized;
private String middleNameInitial;
private String postalCode;
private String rowNumber;
private String state;
private String submissionType;
#DynamoDbPartitionKey
public int getSubmissionId() {
return submissionId;
}
public void setSubmissionId(int submissionId) {
this.submissionId = submissionId;
}
#DynamoDbSortKey
public String getId() {
return id;
}
public void setId(String id) {
this.id = id;
}
public String getAddress1() {
return address1;
}
public void setAddress1(String Address1) {
this.address1 = Address1;
}
public String getCity() {
return city;
}
public void setCity(String city) {
this.city = city;
}
public String getDateOfBirth() {
return dateOfBirth;
}
public void setDateOfBirth(String dateOfBirth) {
this.dateOfBirth = dateOfBirth;
}
public String getErrors() {
return errors;
}
public void setErrors(String errors) {
this.errors = errors;
}
public String getFirstName() {
return firstName;
}
public void setFirstName(String firstName) {
this.firstName = firstName;
}
public String getFirstNameNormalized() {
return firstNameNormalized;
}
public void setFirstNameNormalized(String firstNameNormalized) {
this.firstNameNormalized = firstNameNormalized;
}
public String getGender() {
return gender;
}
public void setGender(String gender) {
this.gender = gender;
}
public String getLastName() {
return lastName;
}
public void setLastName(String lastName) {
this.lastName = lastName;
}
public String getLastNameNormalized() {
return lastNameNormalized;
}
public void setLastNameNormalized(String lastNameNormalized) {
this.lastNameNormalized = lastNameNormalized;
}
public String getMiddleNameInitial() {
return middleNameInitial;
}
public void setMiddleNameInitial(String middleNameInitial) {
this.middleNameInitial = middleNameInitial;
}
public String getPostalCode() {
return postalCode;
}
public void setPostalCode(String postalCode) {
this.postalCode = postalCode;
}
public String getRowNumber() {
return rowNumber;
}
public void setRowNumber(String rowNumber) {
this.rowNumber = rowNumber;
}
public String getState() {
return state;
}
public void setState(String state) {
this.state = state;
}
public String getSubmissionType() {
return submissionType;
}
public void setSubmissionType(String submissionType) {
this.submissionType = submissionType;
}
}
DynamoDB code to get all records
//Connection
DynamoDbClient ddb = DynamoDbClient.builder().build();
DynamoDbEnhancedClient enhancedClient = DynamoDbEnhancedClient.builder()
.dynamoDbClient(ddb)
.build();
//Define table
DynamoDbTable<ErvivePdiDataInvalidQa> table = enhancedClient.table("ervive-pdi-data-invalid-qa", TableSchema.fromBean(ErvivePdiDataInvalidQa.class));
//Get All Items from table - RETURNING NULL
Iterator<ErvivePdiDataInvalidQa> results = table.scan().items().iterator();
while (results.hasNext()) {
ErvivePdiDataInvalidQa rec = results.next();
System.out.println("The record id is "+rec.getId());
}
Also tried:
DynamoDB code to filter by SubmissionID
AttributeValue attr = AttributeValue.builder()
.n("1175")
.build();
// Get only Open items in the Work table
Map<String, AttributeValue> myMap = new HashMap<>();
myMap.put(":val1", attr);
Map<String, String> myExMap = new HashMap<>();
myExMap.put("#sid", "SubmissionId");
// Set the Expression so only Closed items are queried from the Work table
Expression expression = Expression.builder()
.expressionValues(myMap)
.expressionNames(myExMap)
.expression("#sid = :val1")
.build();
ScanEnhancedRequest enhancedRequest = ScanEnhancedRequest.builder()
.filterExpression(expression)
.limit(15)
.build();
// Get items in the Record table and write out the ID value
Iterator<ErvivePdiDataInvalidQa> results = table.scan(enhancedRequest).items().iterator();
while (results.hasNext()) {
ErvivePdiDataInvalidQa record = results.next();
System.out.println("The record id is " + record.getId());
}
Related
I'm having trouble running a query against a secondary index, getting an exception:
Ex getting dynamodb scan: java.lang.IllegalArgumentException: Attempt to execute an operation that requires a secondary index without defining the index attributes in the table metadata. Index name: category-timestamp-index
Can someone guide me on how I'm doing this wrong?
My table is idIT_RSS_Sources and I've created an index category-timestamp-index.
screenshot attached of index
My code is:
DynamoDbEnhancedClient enhancedClient = getEnhancedDBClient(region);
// Create a DynamoDbTable object
logger.debug("getting RSS Source category-timestamp-index");
//this throws the exception
DynamoDbIndex<RSS_Source> catIndex =
enhancedClient.table("idIT_RSS_Sources",
TableSchema.fromBean(RSS_Source.class))
.index("category-timestamp-index");
logger.debug("building query attributes");
AttributeValue att = AttributeValue.builder()
.s(theCategory)
.build();
Map<String, AttributeValue> expressionValues = new HashMap<>();
expressionValues.put(":value", att);
Expression expression = Expression.builder()
.expression("category = :value")
.expressionValues(expressionValues)
.build();
// Create a QueryConditional object that's used in the query operation
QueryConditional queryConditional = QueryConditional
.keyEqualTo(Key.builder().partitionValue(theCategory)
.build());
logger.debug("calling catIndex.query in getRSS...ForCategory");
Iterator<Page<RSS_Source>> dbFeedResults = (Iterator<Page<RSS_Source>>) catIndex.query(
QueryEnhancedRequest.builder()
.queryConditional(queryConditional)
.build());
solved, I was not using the proper annotation in my model class:
#DynamoDbSecondaryPartitionKey(indexNames = { "category-index" })
public String getCategory() { return category; }
public void setCategory(String category) { this.category = category; }
Assume you have a model named Issues.
package com.example.dynamodb;
import software.amazon.awssdk.enhanced.dynamodb.mapper.annotations.DynamoDbBean;
import software.amazon.awssdk.enhanced.dynamodb.mapper.annotations.DynamoDbPartitionKey;
import software.amazon.awssdk.enhanced.dynamodb.mapper.annotations.DynamoDbSecondaryPartitionKey;
import software.amazon.awssdk.enhanced.dynamodb.mapper.annotations.DynamoDbSortKey;
#DynamoDbBean
public class Issues {
private String issueId;
private String title;
private String createDate;
private String description;
private String dueDate;
private String status;
private String priority;
private String lastUpdateDate;
#DynamoDbPartitionKey
public String getId() {
return this.issueId;
}
public void setId(String id) {
this.issueId = id;
}
#DynamoDbSortKey
public String getTitle() {
return this.title;
}
public void setTitle(String title) {
this.title = title;
}
public void setLastUpdateDate(String lastUpdateDate) {
this.lastUpdateDate = lastUpdateDate;
}
public String getLastUpdateDate() {
return this.lastUpdateDate;
}
public void setPriority(String priority) {
this.priority = priority;
}
public String getPriority() {
return this.priority;
}
public void setStatus(String status) {
this.status = status;
}
public String getStatus() {
return this.status;
}
public void setDueDate(String dueDate) {
this.dueDate = dueDate;
}
#DynamoDbSecondaryPartitionKey(indexNames = { "dueDateIndex" })
public String getDueDate() {
return this.dueDate;
}
public String getDate() {
return this.createDate;
}
public void setDate(String date) {
this.createDate = date;
}
public String getDescription() {
return this.description;
}
public void setDescription(String description) {
this.description = description;
}
}
Notice the annotation on getDueDate.
#DynamoDbSecondaryPartitionKey(indexNames = { "dueDateIndex" })
public String getDueDate() {
return this.dueDate;
}
This is because the Issues table has a secondary index named dueDateIndex.
To query on this secondary index, you can use this code that uses the Amazon DynamoDB Java API V2:
public static void queryIndex(DynamoDbClient ddb, String tableName, String indexName) {
try {
// Create a DynamoDbEnhancedClient and use the DynamoDbClient object
DynamoDbEnhancedClient enhancedClient = DynamoDbEnhancedClient.builder()
.dynamoDbClient(ddb)
.build();
//Create a DynamoDbTable object based on Issues
DynamoDbTable<Issues> table = enhancedClient.table("Issues", TableSchema.fromBean(Issues.class));
String dateVal = "2013-11-19";
DynamoDbIndex<Issues> secIndex =
enhancedClient.table("Issues",
TableSchema.fromBean(Issues.class))
.index("dueDateIndex");
AttributeValue attVal = AttributeValue.builder()
.s(dateVal)
.build();
// Create a QueryConditional object that's used in the query operation
QueryConditional queryConditional = QueryConditional
.keyEqualTo(Key.builder().partitionValue(attVal)
.build());
// Get items in the Issues table
SdkIterable<Page<Issues>> results = secIndex.query(
QueryEnhancedRequest.builder()
.queryConditional(queryConditional)
.build());
AtomicInteger atomicInteger = new AtomicInteger();
atomicInteger.set(0);
results.forEach(page -> {
Issues issue = (Issues) page.items().get(atomicInteger.get());
System.out.println("The issue title is "+issue.getTitle());
atomicInteger.incrementAndGet();
});
} catch (DynamoDbException e) {
System.err.println(e.getMessage());
System.exit(1);
}
}
For what it's worth, if your Global Secondary Index has a sort key, you must annotate that field in the DynamoDB bean with:
#DynamoDbSecondarySortKey(indexNames = { "<indexName>" })
public String getFieldName() {
return fieldName;
}
My working code is as below:
sortKey-index = GSI in dynamo db
List<Flow> flows = new ArrayList<>();
DynamoDbIndex<Flow> flowBySortKey = table().index("sortKey-index");
// Create a QueryConditional object that's used in the query operation
QueryConditional queryConditional = QueryConditional
.keyEqualTo(Key.builder()
.partitionValue(sortKey)
.build());
SdkIterable<Page<Flow>> dbFeedResults = flowBySortKey.query(
QueryEnhancedRequest.builder()
.queryConditional(queryConditional)
.build());
dbFeedResults.forEach(flowPage -> {
flows.addAll(flowPage.items());
});
Lets say I have 100 documents with fields
Name
Age
Address
Now suppose my business model is change and I want to add new field call PhoneNumber.
How to add field PhoneNumber in all 100 documents ?
Is is possible to such stuff on NoSQL database?
You will have to write code to iterate all the documents to update, then actually update a new value in each one of them. Firestore has no similar command as "update tablename set x=y where ..." in SQL.
Is is possible to such stuff on NoSQL database?
Yes it is! Assuming you have a User model class that look like this:
public class User {
private String name;
private int age;
private String address;
private String phoneNumber; //Property that is newly added
public User() {}
public User(String name, int age, String address, String phoneNumber) {
this.name = name;
this.age = age;
this.address = address;
this.phoneNumber = phoneNumber;
}
public String getName() { return name; }
public void setName(String name) { this.name = name; }
public int getAge() { return age; }
public void setAge(int age) { this.age = age; }
public String getAddress() { return address; }
public void setAddress(String address) { this.address = address; }
public String getPhoneNumber() { return phoneNumber; }
public void setPhoneNumber(String phoneNumber) { this.phoneNumber = phoneNumber; }
}
To actually add a new property and update it accordingly, you need to use setters. If you are setting the values directly onto the public fields, the setters are not mandatory.
How to add field PhoneNumber in all 100 documents?
As also #Doug Stevenson mentioned in his answer, to solve this, you need to iterate all the documents within your users collection. So please use the following lines of code:
db.collection("users").get().addOnCompleteListener(new OnCompleteListener<QuerySnapshot>() {
#Override
public void onComplete(#NonNull Task<QuerySnapshot> task) {
if (task.isSuccessful()) {
for (DocumentSnapshot document : task.getResult()) {
User user = document.toObject(User.class);
user.setPhoneNumber("+1-111-111-111"); //Use the setter
String id = document.getId();
db.collection("users").document(id).set(user); //Set user object
}
}
}
});
The result of this code would be to add the phoneNumber property to all you User objects with a default value of +1-111-111-111. You can also set the value to null if it's more convenient for you. At the end, the updated object is set right on the corresponding reference.
If you are not using a model class, please see my answer from this post.
I am trying to use DynamodbMapper to query data using gsi.
HashMap<String, AttributeValue> eav = new HashMap<>();
eav.put(":v1", new AttributeValue().withS(employee.getDepartment()));
eav.put(":v2", new AttributeValue().withS(employee.getContactId()));
DynamoDBQueryExpression<Employee> queryExpression =
new DynamoDBQueryExpression()
.withIndexName("DepartmentContactId-index")
.withKeyConditionExpression("Department = :v1 and contactId = :v2")
.withExpressionAttributeValues(eav)
.withConsistentRead(false);
List<Employee> items =
dynamoDBMapper.query(Employee.class, queryExpression);
I am getting bad signature exception.
PS: one of the field(column) in Employee table in dynamodb is encrypted using AWSKMS. I have configured the same KMS key in dynamodb mapper but still getting the same issue. Any pointers?
Mapper class -->
package com.test.model;
import com.amazonaws.services.dynamodbv2.datamodeling.*;
importcom.amazonaws.services.dynamodbv2.datamodeling.encryption.DoNotEncrypt;
import static com.test.util.Constants.*;
#DynamoDBTable(tableName = "Employee")
public class Employee {
private String id;
private String department;
private String contactId;
private RulesData rulesData;
// Partition Key
#DynamoDBHashKey(attributeName = ID)
#DynamoDBAutoGeneratedKey
public String getId() {
return id;
}
public void setId(String id) {
this.id = id;
}
#DoNotEncrypt
#DynamoDBRangeKey(attributeName = DEPARTMENT)
public String getDepartment() {
return department;
}
public void setDepartment(String department) {
this.department = department;
}
#DoNotEncrypt
#DynamoDBAttribute(attributeName = CONTACT_ID)
public String getContactId() {
return contactId;
}
public void setContactId(String contactId) {
this.contactId = contactId;
}
#DynamoDBAttribute(attributeName = DATA)
public RulesData getRulesData() {
return rulesData;
}
public void setRulesData(RulesData rulesData) {
this.rulesData = rulesData;
}
}
If you set the projection type of the global secondary index(GSI) to be other than ALL, then the signature attribute will not be in the GSI .
Thus, if you require only unencrypted fields from your query on the GSI, use a new DynamoDBMapper without an AttributeEncryptor.
If you require encrypted fields too, set the projection type of the GSI to be ALL.
I hade a strange happening today with my Firebase project.
Suddenly the
AddressChatMessage chatMessage = snapshot.getValue(AddressChatMessage.class);
parsed everything ok for the AddressChatMessage except for one int.
Took me 2 hour until i tried this, setting the field to public
public int type;
Note I use this code for weeks without problem and today Android studio made some core updated to 2.3.2 and maybe that trigger this strange event.
Here´s the AddressChatMessage.java nothing strange except that the public int type cannot be private, if it is, it will be zero, that too is strange, usually Firebase give out a logcat warning when pojo parsing fails. I have 10 other modell classes like this with plenty of int´s
#IgnoreExtraProperties
public class AddressChatMessage {
// [START Firebase keys inside AddressChatMessage ]
#Exclude
public static final String TYPE = "type";
#Exclude
public static final String SENDER_ID = "senderId";
#Exclude
public static final String MESSAGE = "message";
#Exclude
public static final String FILENAME = "fileName";
#Exclude
public static final String DOWNLOAD_URI = "downloadUri";
#Exclude
public static final String TIME = "time";
// [STOP Firebase keys inside AddressChatMessage ]
public int type;
private String senderId;
private String message;
private String fileName;
private String downloadUri;
#Exclude
private long time;
#Exclude
private String messageId;
public AddressChatMessage() {
}
public AddressChatMessage(int type, String senderUid) {
this.senderId = senderUid;
this.type = type;
}
public AddressChatMessage(int type, String senderUid, String message) {
this.type = type;
this.senderId = senderUid;
this.message = message;
}
private int getType() {
return type;
}
public void setType(int type) {
this.type = type;
}
public String getSenderId() {
return senderId;
}
public void setSenderId(String senderId) {
this.senderId = senderId;
}
public String getMessage() {
return message;
}
public void setMessage(String message) {
this.message = message;
}
public String getFileName() {
return fileName;
}
public void setFileName(String fileName) {
this.fileName = fileName;
}
public String getDownloadUri() {
return downloadUri;
}
public void setDownloadUri(String downloadUri) {
this.downloadUri = downloadUri;
}
public long getTime() {
return time;
}
public String getMessageId() {
return messageId;
}
public void setMessageId(String messageId) {
this.messageId = messageId;
}
#Exclude
public Map<String, Object> toMap() {
HashMap<String, Object> result = new HashMap<>();
result.put(TYPE, type);
result.put(SENDER_ID, senderId);
result.put(MESSAGE, message);
result.put(FILENAME, fileName);
result.put(DOWNLOAD_URI, downloadUri);
result.put(TIME, time);
return result;
}
#Exclude
public boolean isTypeNormal() {
return getType() == ChatAdapter.MessageType.NORMAL.ordinal();
}
#Exclude
public boolean isTypeImage() {
return getType() == ChatAdapter.MessageType.IMAGE.ordinal();
}
}
When cleaning the code using Lint. Lint suggested changes like "This can be private instead of public" - I accidentally set the getType() to private access.
How do I use the JdbcTemplate.query()/queryForList() to run a query using namedParameter and store the result set into a List of 'User's?
User Class:
public class User {
String name = null;
String id = null;
public String getName() {
return name;
}
public void setName(String name) {
this.name = name;
}
public String getId() {
return name;
}
public void setId(String id) {
this.id = id;
}
}
Query:
SELECT name, id FROM USERS where email=:email
I'm looking for something like:
ArrayList<User> userList = jdbcTemplate.query(sql_query,
...some_mapper..., etc);
Seems like the answer to the question is not available at one place, on the Internet. Here's what I found out:
For adding the resultset into a List<>, we can use the NamedParameterJdbcTemplate.query() function:
NamedParameterJdbcTemplate jdbcTemplate;
ArrayList<User> usersSearchResult = (ArrayList<User>) jdbcTemplate.query(
USER_LIST_TP_query,
namedParameters,
new RowMapperResultSetExtractor<User>(new UserRowMapper(), 20));
We also have to define a custom RowMapperResultSetExtractor so that JDBC can understand how to convert each row in the result set to the type User.
private class UserRowMapper implements RowMapper<User> {
public User mapRow(ResultSet rs, int rowNum) throws SQLException {
User user = new User();
user.setId(rs.getString("ID"));
user.setName(rs.getString("NAME"));
return user;
}
}