I have a table as : AdvgCountries which has two columns
a. CountryId (String) (Parition Key)
b. CountryName(String) Sort Key
While creating the table , I created with only Partition Key and then later added a Global Secondary Index with Index name as:
CountryName-index
Type : GSI
Partition key : CountryId
Sort Key : CountryName
I am able to retrieve CountryName based upon CountryId but unable to retrieve CountryId based upon CountryName. Based upon my reading I found that there are options to do this by providing indexname but I get the following error:
botocore.exceptions.ClientError: An error occurred
(ValidationException) when calling the Query operation: Query
condition missed key schema element: CountryId
import boto3
import json
import os
from boto3.dynamodb.conditions import Key, Attr
def query_bycountryname(pCountryname, dynamodb=None):
if not dynamodb:
dynamodb = boto3.resource('dynamodb', endpoint_url="https://dynamodb.us-east-1.amazonaws.com")
table = dynamodb.Table('AdvgCountires')
print(f"table")
attributes = table.query(
IndexName="CountryName-index",
KeyConditionExpression=Key('CountryName').eq(pCountryname),
)
if 'Items' in attributes and len(attributes['Items']) == 1:
attributes = attributes['Items'][0]
print(f"before return")
return attributes
if __name__ == '__main__':
CountryName = "India"
print(f"Data for {CountryName}")
countries = query_bycountryname(CountryName)
for country in countries:
print(country['CountryId'], ":", country['CountryName'])
Any help is appreciated.
You can't be able to fetch primary key value based on sort key. DynamoDB does not work like this.
In Dynamodb, each item’s location is determined by the hash value of
its partition key.
The Query operation in Amazon DynamoDB finds items based on primary
key values.
KeyConditionExpression are used to write conditional statements by
using comparison operators that evaluate against a key and limit the
items returned. In other words, you can use special operators to
include, exclude, and match items by their sort key values.
Related
I have tried creating GSI with a PK that uses a composite value of business_id, type_id, partner_id fields. I did it in two different ways in the AWS console:
First: business_id#type_id#partner_id
Second: [business_id]#[type_id]#[partner_id]
and sort key: updated
Here is the query:
SELECT *
FROM "items"."composite_key-index"
WHERE business_id = 435634652 AND type_id = 2 AND partner_id = 69992528
ORDER BY updated ASC
In both cases it throws this error:
ValidationException: Must have at least one non-optional hash key
condition in WHERE clause when using ORDER BY clause.
And if I run it without the order by:
SELECT *
FROM "items"."composite_key-index"
WHERE business_id = 435634652 AND type_id = 2 AND partner_id = 69992528
it doesnt return any items, even though there are data matching those values.
What am I doing wrong here?
To use a composite value as a key, you have to build the values yourself.
Your application would have to store the value in a single attribute, ex GSI_PK, as 435634652#2#6992528
Then your query would look like
SELECT *
FROM "items"."composite_key-index"
WHERE GSI_PK = "435634652#2#6992528"
Table : Customer
Item: CustomerId,PurchaseType,Name,mobilenumber,price, createdDate
DATA1: cus001,"online","BBBBB","yourmobilenumber",6000,"01/07/2017 01:12:05"
DATA2: cus002,"online","myname","mymobilenumber",500,"10/07/2017 01:12:01"
DATA3: cus003,"online","AAAAA","yourmobilenumber",6000,"10/07/2017 01:12:06"
DATA4: cus004,"online","yourname","yourmobilenumber",1000,"10/07/2017 02:12:06"
DATA5: cus005,"retail","yourname","yourmobilenumber",1000,"10/07/2017 03:12:06"
GSI: price-index[PurchaseType,price]
Query with index "price-index"
condition: purchasetype="online" and price >500
ScanIndex: true
How to get the result based on the following conditions:
purchasetype="online"
price>500
order by Name
You need to create a different GSI:
PurchaseType - partition key of the GSI
Name - sort key of the GSI
Then you can use a query to find all items with the necessary purchase type, order by Name and provide a filter expression to filter all items with high prices.
I'm familiar with MySQL and am starting to use Amazon DynamoDB for a new project.
Assume I have a MySQL table like this:
CREATE TABLE foo (
id CHAR(64) NOT NULL,
scheduledDelivery DATETIME NOT NULL,
-- ...other columns...
PRIMARY KEY(id),
INDEX schedIndex (scheduledDelivery)
);
Note the secondary Index schedIndex which is supposed to speed-up the following query (which is executed periodically):
SELECT *
FROM foo
WHERE scheduledDelivery <= NOW()
ORDER BY scheduledDelivery ASC
LIMIT 100;
That is: Take the 100 oldest items that are due to be delivered.
With DynamoDB I can use the id column as primary partition key.
However, I don't understand how I can avoid full-table scans in DynamoDB. When adding a secondary index I must always specify a "partition key". However, (in MySQL words) I see these problems:
the scheduledDelivery column is not unique, so it can't be used as a partition key itself AFAIK
adding id as unique partition key and using scheduledDelivery as "sort key" sounds like a (id, scheduledDelivery) secondary index to me, which makes that index pratically useless
I understand that MySQL and DynamoDB require different approaches, so what would be a appropriate solution in this case?
It's not possible to avoid a full table scan with this kind of query.
However, you may be able to disguise it as a Query operation, which would allow you to sort the results (not possible with a Scan).
You must first create a GSI. Let's name it scheduled_delivery-index.
We will specify our index's partition key to be an attribute named fixed_val, and our sort key to be scheduled_delivery.
fixed_val will contain any value you want, but it must always be that value, and you must know it from the client side. For the sake of this example, let's say that fixed_val will always be 1.
GSI keys do not have to be unique, so don't worry if there are two duplicated scheduled_delivery values.
You would query the table like this:
var now = Date.now();
//...
{
TableName: "foo",
IndexName: "scheduled_delivery-index",
ExpressionAttributeNames: {
"#f": "fixed_value",
"#d": "scheduled_delivery"
},
ExpressionAttributeValues: {
":f": 1,
":d": now
},
KeyConditionExpression: "#f = :f and #d <= :d",
ScanIndexForward: true
}
I am having trouble using AWS Boto3 to query DynamoDB with a hash key and a range key at the same time using the recommend KeyConditionExpression. I have attached an example query:
import boto3
from boto3 import dynamodb
from boto3.session import Session
dynamodb_session = Session(aws_access_key_id=AWS_KEY,
aws_secret_access_key=AWS_PASS,
region_name=DYNAMODB_REGION)
dynamodb = dynamodb_session.resource('dynamodb')
table=dynamodb.Table(TABLE_NAME)
request = {
'ExpressionAttributeNames': {
'#n0': 'hash_key',
'#n1': 'range_key'
},
'ExpressionAttributeValues': {
':v0': {'S': MY_HASH_KEY},
':v1': {'N': GT_RANGE_KEY}
},
'KeyConditionExpression': '(#n0 = :v0) AND (#n1 > :v1)',
'TableName': TABLE_NAME
}
response = table.query(**request)
When I run this against a table with the following scheme:
Table Name: TABLE_NAME
Primary Hash Key: hash_key (String)
Primary Range Key: range_key (Number)
I get the following error and I cannot understand why:
ClientError: An error occurred (ValidationException) when calling the Query operation: Invalid KeyConditionExpression: Incorrect operand type for operator or function; operator or function: >, operand type: M
From my understanding the type M would be a map or dictionary type and I am using a type N which is a number type and matches my table scheme for the range key. If someone could explain why this error is happening or I am also open to a different way of accomplishing the same query even if you cannot explain why this error exists.
The Boto 3 SDK constructs a Condition Expression for you when you use the Key and Attr functions imported from boto3.dynamodb.conditions:
response = table.query(
KeyConditionExpression=Key('hash_key').eq(hash_value) & Key('range_key').eq(range_key_value)
)
Reference: Step 4: Query and Scan the Data
Hope it helps
Adding this solution as the accepted answer did not address why the query used did not work.
TLDR: Using query on a Table resource in boto3 has subtle differences as opposed to using client.query(...) and requires a different syntax.
The syntax is valid for a query on a client, but not on a Table. The ExpressionAttributeValues on a table do not require you to specify the data type. Also if you are executing a query on a Table resource you do not have to specify the TableName again.
Working solution:
from boto3.session import Session
dynamodb_session = Session(aws_access_key_id=AWS_KEY,aws_secret_access_key=AWS_PASS,region_name=DYNAMODB_REGION)
dynamodb = dynamodb_session.resource('dynamodb')
table = dynamodb.Table(TABLE_NAME)
request = {
'ExpressionAttributeNames': {
'#n0': 'hash_key',
'#n1': 'range_key'
},
'ExpressionAttributeValues': {
':v0': MY_HASH_KEY,
':v1': GT_RANGE_KEY
},
'KeyConditionExpression': '(#n0 = :v0) AND (#n1 > :v1)',
}
response = table.query(**request)
I am the author of a package called botoful which might be useful to avoid dealing with these complexities. The code using botoful will be as follows:
import boto3
from botoful import Query
client = boto3.Session(
aws_access_key_id=AWS_KEY,
aws_secret_access_key=AWS_PASS,
region_name=DYNAMODB_REGION
).client('dynamodb')
results = (
Query(TABLE_NAME)
.key(hash_key=MY_HASH_KEY, range_key__gt=GT_RANGE_KEY)
.execute(client)
)
print(results.items)
In SQLite I can run the following query to get a list of columns in a table:
PRAGMA table_info(myTable)
This gives me the columns but no information about what the primary keys may be. Additionally, I can run the following two queries for finding indexes and foreign keys:
PRAGMA index_list(myTable)
PRAGMA foreign_key_list(myTable)
But I cannot seem to figure out how to view the primary keys. Does anyone know how I can go about doing this?
Note: I also know that I can do:
select * from sqlite_master where type = 'table' and name ='myTable';
And it will give the the create table statement which shows the primary keys. But I am looking for a way to do this without parsing the create statement.
The table_info DOES give you a column named pk (last one) indicating if it is a primary key (if so the index of it in the key) or not (zero).
To clarify, from the documentation:
The "pk" column in the result set is zero for columns that are not
part of the primary key, and is the index of the column in the primary
key for columns that are part of the primary key.
Hopefully this helps someone:
After some research and pain the command that worked for me to find the primary key column name was:
SELECT l.name FROM pragma_table_info("Table_Name") as l WHERE l.pk = 1;
For the ones trying to retrieve a pk name in android, and while using the ROOM library.
#Oogway101's answer was throwing an error: "no such column [your_table_name] ... etc.. etc...
my way of query submition was:
String pkSearch = "SELECT l.name FROM pragma_table_info(" + tableName + ") as l WHERE l.pk = 1;";
database.query(new SimpleSQLiteQuery(pkSearch)
I tried using the (") quotations and still error.
String pkSearch = "SELECT l.name FROM pragma_table_info(\"" + tableName + "\") as l WHERE l.pk = 1;";
So my solution was this:
String pragmaInfo = "PRAGMA table_info(" + tableName + ");";
Cursor c = database.query(new SimpleSQLiteQuery(pragmaInfo));
String id = null;
c.moveToFirst();
do {
if (c.getInt(5) == 1) {
id = c.getString(1);
}
} while (c.moveToNext() && id == null);
Log.println(Log.ASSERT, TAG, "AbstractDao: pk is: " + id);
The explanation is that:
A) PRAGMA table_info returns a cursor with various indices, the response is atleast of length 6... didnt check more...
B) index 1 has the column name.
C) index 5 has the "pk" value, either 0 if it is not a primary key, or 1 if its a pk.
You can define more than one pk so this will not bring an accurate result if your table has more than one (IMHO more than one is bad design and balloons the complexity of the database beyond human comprehension).
So how will this fit into the #Dao? (you may ask...)
When making the Dao "abstract" you have access to a default constructor which has the database in it:
from the docummentation:
An abstract #Dao class can optionally have a constructor that takes a Database as its only parameter.
this is the constructor that will grant you access to the query.
There is a catch though...
You may use the Dao during a database creation with the .addCallback() method:
instance = Room.databaseBuilder(context.getApplicationContext(),
AppDatabase2.class, "database")
.addCallback(
//You may use the Daos here.
)
.build();
If you run a query in the constructor of the Dao, the database will enter a feedback loop of infinite instantiation.
This means that the query MUST be used LAZILY (just at the moment the user needs something), and because the value will never change, it can be stored. and never re-queried.