Let's assume my table looks like:
Code |StartDate |EndDate |Additional Attributes...
ABC |11-24-2015 |11-26-2015 | ....
ABC |12-12-2015 |12-15-2015 | ....
ABC |10-05-2015 |10-10-2015 | ....
PQR |03-24-2015 |03-27-2015 | ....
PQR |05-04-2015 |05-08-2015 | ....
Provided a Code (c) and a date range (x, y), I need to be able to query items something like:
Query => (Code = c) AND ((StartDate BETWEEN x AND y) OR (EndDate BETWEEN x AND y))
I was planning to use a Primary Key as a Hash and Range Key (Code, StartDate) with an additional LSI (EndDate) and do a query on it.
I am not sure if there is a way to achieve this. I don't want to use the SCAN operation as it seems to scan the entire table which could be very costly.
Also, would like to achieve this in a single query.
One option would be to do this using QUERY and a FilterExpression. No need to define the LSI on this case. You would have to query by Hash Key with the EQ operator and then narrow the results with the Filter Expression. Here is an example with the Java SDK:
Table table = dynamoDB.getTable(tableName);
Map<String, Object> expressionAttributeValues = new HashMap<String, Object>();
expressionAttributeValues.put(":x", "11-24-2015");
expressionAttributeValues.put(":y", "11-26-2015");
QuerySpec spec = new QuerySpec()
.withHashKey("Code", "CodeValueHere")
.withFilterExpression("(StartDate between :x and :y) or (EndDate between :x and :y)")
.withValueMap(expressionAttributeValues);
ItemCollection<QueryOutcome> items = table.query(spec);
Iterator<Item> iterator = items.iterator();
while (iterator.hasNext()) {
System.out.println(iterator.next().toJSONPretty());
}
See Specifying Conditions with Condition Expressions for more details.
Additionally, although the previous query only uses the Hash Key , you can still group the records with the Range Key containing the dates in the following format:
StartDate#EndDate
Table Structure:
Code DateRange |StartDate |EndDate
ABC 11-24-2015#11-26-2015 |11-24-2015 |11-26-2015
ABC 12-12-2015#12-15-2015 |12-12-2015 |12-15-2015
ABC 10-05-2015#10-10-2015 |10-05-2015 |10-10-2015
PQR 03-24-2015#03-27-2015 |03-24-2015 |03-27-2015
PQR 05-04-2015#05-08-2015 |05-04-2015 |05-08-2015
This way If you happen to query only by Hash Key you would still get the records sorted by the dates. Also, I believe it is a good idea to follow the advice given about the unambiguous date format.nu
Related
Is there any way of supplying multiple values for a DynamoDB table's Sort Key whilst doing a query in Boto3?
For a single SK value to search on, I'm doing this:
table.query(
IndexName="my_gsi",
KeyConditionExpression=Key('my_gsi_pk').eq({pk value}) & Key('my_gsi_sk').eq({sk value}),
FilterExpression={filter expression}
)
... which works.
However, my scenario involves searching on one of a couple of potential SK values, so I'd like to, in SQL terms, do something like this:
WHERE my_gsi_pk = {pk value}
AND my_gsi_sk IN ({sk value 1}, {sk value 2})
I've looked in the Boto3 documentation in the .query() section and concentrated upon the KeyConditionExpression syntax but can't identify whether this is possible or not.
The query API does not support the IN operator in the KeyConditionExpression.
Use the execute_statement API instead. This executes a PartiQL statement, which does accept the IN operator in query operations for the Partition and Sort keys:
sk = ["Foo", "Bar"]
res = client.execute_statement(
Statement=f'SELECT * FROM "my_table"."my_gsi" WHERE my_gsi_pk = ? AND my_gsi_sk IN [{",".join(["?" for k in sk])}]',
Parameters= [{"S": "1"}] + [{"S": k} for k in sk]
)
This creates a PartiQL Statement like SELECT * FROM "my_table"."my_gsi" WHERE my_gsi_pk = ? AND my_gsi_sk IN [?, ?] and substitution Parameters like [{"S": "1"}, {"S": "Foo"}, {"S": "Bar"}].
Please note that the PartiQL will spend much more RCU than the Query. You can check this by requesting ReturnConsumedCapacity = ReturnConsumedCapacity.TOTAL
I have a need to be able to query Azure Data Explorer (ADX) tables dynamically, that is, using application-specific metadata that is also stored in ADX.
If this is even possible, the way to do it seems to be via the table() function. In other words, it feels like I should be able to simply write:
let table_name = <non-trivial ADX query that returns the name of a table as a string>;
table(table_name) | limit 10
But this query fails since I am trying to pass a variable to the table() function, and "a parameter, which is not scalar constant string can't be passed as parameter to table() function". The workaround provided doesn't really help, since all the possible table names are not known ahead of time.
Is there any way to do this all within ADX (i.e. without multiple queries from the client) or do I need to go back to the drawing board?
if you know the desired output schema, you could potentially achieve that using union (note that in this case, the result schema will be the union of all tables, and you'll need to explicitly project the columns you're interested in)
let TableA = view() { print col1 = "hello world"};
let TableB = view() { print col1 = "goodbye universe" };
let LabelTable = datatable(table_name:string, label:string, updated:datetime)
[
"TableA", "MyLabel", datetime(2019-10-08),
"TableB", "MyLabel", datetime(2019-10-02)
];
let GetLabeledTable = (l:string)
{
toscalar(
LabelTable
| where label == l
| order by updated desc
| limit 1
)
};
let table_name = GetLabeledTable('MyLabel');
union withsource = T *
| where T == table_name
| project col1
In SQLite I can run the following query to get a list of columns in a table:
PRAGMA table_info(myTable)
This gives me the columns but no information about what the primary keys may be. Additionally, I can run the following two queries for finding indexes and foreign keys:
PRAGMA index_list(myTable)
PRAGMA foreign_key_list(myTable)
But I cannot seem to figure out how to view the primary keys. Does anyone know how I can go about doing this?
Note: I also know that I can do:
select * from sqlite_master where type = 'table' and name ='myTable';
And it will give the the create table statement which shows the primary keys. But I am looking for a way to do this without parsing the create statement.
The table_info DOES give you a column named pk (last one) indicating if it is a primary key (if so the index of it in the key) or not (zero).
To clarify, from the documentation:
The "pk" column in the result set is zero for columns that are not
part of the primary key, and is the index of the column in the primary
key for columns that are part of the primary key.
Hopefully this helps someone:
After some research and pain the command that worked for me to find the primary key column name was:
SELECT l.name FROM pragma_table_info("Table_Name") as l WHERE l.pk = 1;
For the ones trying to retrieve a pk name in android, and while using the ROOM library.
#Oogway101's answer was throwing an error: "no such column [your_table_name] ... etc.. etc...
my way of query submition was:
String pkSearch = "SELECT l.name FROM pragma_table_info(" + tableName + ") as l WHERE l.pk = 1;";
database.query(new SimpleSQLiteQuery(pkSearch)
I tried using the (") quotations and still error.
String pkSearch = "SELECT l.name FROM pragma_table_info(\"" + tableName + "\") as l WHERE l.pk = 1;";
So my solution was this:
String pragmaInfo = "PRAGMA table_info(" + tableName + ");";
Cursor c = database.query(new SimpleSQLiteQuery(pragmaInfo));
String id = null;
c.moveToFirst();
do {
if (c.getInt(5) == 1) {
id = c.getString(1);
}
} while (c.moveToNext() && id == null);
Log.println(Log.ASSERT, TAG, "AbstractDao: pk is: " + id);
The explanation is that:
A) PRAGMA table_info returns a cursor with various indices, the response is atleast of length 6... didnt check more...
B) index 1 has the column name.
C) index 5 has the "pk" value, either 0 if it is not a primary key, or 1 if its a pk.
You can define more than one pk so this will not bring an accurate result if your table has more than one (IMHO more than one is bad design and balloons the complexity of the database beyond human comprehension).
So how will this fit into the #Dao? (you may ask...)
When making the Dao "abstract" you have access to a default constructor which has the database in it:
from the docummentation:
An abstract #Dao class can optionally have a constructor that takes a Database as its only parameter.
this is the constructor that will grant you access to the query.
There is a catch though...
You may use the Dao during a database creation with the .addCallback() method:
instance = Room.databaseBuilder(context.getApplicationContext(),
AppDatabase2.class, "database")
.addCallback(
//You may use the Daos here.
)
.build();
If you run a query in the constructor of the Dao, the database will enter a feedback loop of infinite instantiation.
This means that the query MUST be used LAZILY (just at the moment the user needs something), and because the value will never change, it can be stored. and never re-queried.
I've just started my adventure with Cassandra database. I've managed to learn some basics but what I still can't understand is how to work with dates in Cassandra?
So for example in MySQL we have a datetime type for a field and we can query (for example) all fields with creation date less then 2010-01-01. Furthermore we can order the result by creation date field.
How can we achieve the same with Cassandra? How to define the corresponding Column Family and how to query (CQL) it to get the same result?
You can use type DateType to define a column of type DateType in your column family. You should really read this page, it has description and example how to do range query (that is creationdate < 2010-01-01). For ordering, you can refer to the SliceRange but this will probably cover in the cassandra client already. You will probably want to look into the cassandra client to do the query.
This is a snippet on how to do query in cassandra using hector client.
// 2010-01-01
Date date = new Date(1262275200L);
try
{
getConnection();
IndexedSlicesQuery<String, String, String> indexedSlicesQuery = HFactory.createIndexedSlicesQuery(keyspace, ss, ss, ss);
indexedSlicesQuery.setColumnNames("name");
indexedSlicesQuery.addLtExpression("timestamp", ByteBufferUtil.string(date_s.toByteBuffer(date)));
indexedSlicesQuery.addEqualsExpression("searchall", ByteBufferUtil.string(bs.toByteBuffer(true)));
indexedSlicesQuery.setColumnFamily(column_family);
indexedSlicesQuery.setStartKey("");
System.out.println(indexedSlicesQuery.toString());
QueryResult<OrderedRows<String, String, String>> res = indexedSlicesQuery.execute();
List<Row<String, String, String>> list = res.get().getList();
for (Row<?, ?, ?> row : list)
{
System.out.println(row.getKey());
}
}
Using SQL Server, have a name value pair table. Each row is basically userid, contentid, sectionid, parameter, value. So there is data I want to display in a table such as user information. Each bit of information is in it's own row, sow how do I get it into a DataSet for use in a Repeater? Can I somehow merge the rows into one? So I can get multiple parameter/value pairs on one row?
so like...
two rows for user 32:
(param / value)
fname / Bob
lname / Smith
displayed on one row in a repeater like this:
Bob Smith
Any ideas? Oh yeah and the reason it is in the name/value pair format is to adhere to a required standard.
Maybe something like...
SELECT fff.firstname, lll.lastname
FROM (
SELECT ff.value AS firstname
FROM PairTable AS ff
WHERE ff.param = 'fname'
AND ff.userId = 32
) fff, (
SELECT ll.value AS lastname
FROM PairTable AS ll
WHERE ll.param = 'lname'
AND ll.userId = 32
) lll
Sucky standard.... :)
Anyway, your best bet is to play some magic (cursor) with your stored proc and return the data from the stored procedure in the format you want. Then binding to a DataSet, or a string list, is trivial.
Another alternative is PIVOT.
Something like this (untested because I have no SQL Server around now)
SELECT UserID, [fname] AS firstname, [lname] AS lastname
FROM
(SELECT UserID, value, name from PairTable WHERE UserID=32 ) p
PIVOT
(
value
FOR name IN
( [fname], [lname] )
) AS pvt
That table is awful. No offense ;-)
Relational databases just doen't do EAV tables very well.
You can also group, and do conditional CASE statements - like this:
SELECT UserID,
MAX(CASE WHEN param = 'fname' THEN value ELSE '' END) AS fname,
MAX(CASE WHEN param = 'lname' THEN value ELSE '' END) AS lname
FROM MyEAVTable
GROUP BY UserID
The PIVOT syntax is great - the only benefit to this solution is that it will work with SQL 2000 as well.