I have a QODBC query for QuickBooks that generates Profit & Loss report with dynamic columns. The parameterized date value works to return the requested three daily values, AND 300+ more columns of None!
sql1 = textwrap.dedent("""
sp_report
ProfitAndLossStandard
show
"AccountNumber",
"Text",
"Label",
"RowType",
"Amount" <-- dynamic
parameters
DateFrom=?,
DateTo=?,
ReportBasis='Accrual',
SummarizeColumnsBy='Day',
ReturnRows='All'
""")
cursor.execute(sql1, date(2022,11,1), date(2022,11,3))
for row in cursor.fetchall():
print(row)
The first row looks like this:
(None, 'Ordinary Income/Expense', None, 'TextRow', 366, None, None, None, None, None, None, None, None, ... (300+), None)
But if I declare the date parameter using the ODBC escape sequence:
parameters
DateFrom={d'2022-11-01'},
DateTo={d'2022-11-03'},
Then, I get the sensible results:
(None, 'Ordinary Income/Expense', None, 'TextRow', 4, None, None, None, None)
Why does my PyODBC parameterized date query return too much data?
Bonus: See the additional column (not named in the query) which shows 366 in the first query and 4 in the second--What is this?
Note: The None values are not significant to the problem. It's the number of columns per row that's the issue.
Related
I'm currently experimenting with Airflow for monitoring tasks regarding Snowflake and I'd like to execute a simple DAG with one task that pushes a SQL query to in Snowflake and should check the returned value that should be a number to be greater than a defined threshold.
So the following is basically my sql Statement in the DAG definition:
query_check = """select COUNT(*)
FROM (select CASE WHEN NAME LIKE '%SW_PRODUCTFEED%' THEN 'PRODUCTFEED'
ELSE NULL END AS TASKTREE_NAME
, NAME
, STATE
, ERROR_MESSAGE
, SCHEDULED_TIME
, QUERY_START_TIME
, NEXT_SCHEDULED_TIME
from table(TEST_DB.INFORMATION_SCHEMA.task_history())
where TASKTREE_NAME IS NOT NULL
qualify DENSE_RANK() OVER (PARTITION BY TASKTREE_NAME ORDER BY to_date(SCHEDULED_TIME) desc) < 3
order by scheduled_time desc);"""
Then the following is the definition of the DAG and the task within it:
with dag:
query1_exec = SnowflakeCheckOperator(
task_id="snowflake_check_task_history",
sql=query_check,
params={
"check_name": "number_rows",
"check_statement": "count >=1"
},
conn_id="Snowflake_test"
)
query1_exec
I'd like to use the SnowflakeCheckOperator to check the returned value from the query if it's greater than 1
However, it seems that Snowflake or the SnowflakeOperator in that case is returning the result of the query in a dict object, like so:
Record: {'COUNT(*)': 10}
Therefore the check always results in a true statement because the SnowflakeCheckOperator isn't checking against the value of the Record["Count"] but something else.
Now my question is how to handle the return value so that the check is evaluated against right value? Is it possible to change the format of the return value? Or maybe get access to the value of the key of the dict object?
Is there any way of supplying multiple values for a DynamoDB table's Sort Key whilst doing a query in Boto3?
For a single SK value to search on, I'm doing this:
table.query(
IndexName="my_gsi",
KeyConditionExpression=Key('my_gsi_pk').eq({pk value}) & Key('my_gsi_sk').eq({sk value}),
FilterExpression={filter expression}
)
... which works.
However, my scenario involves searching on one of a couple of potential SK values, so I'd like to, in SQL terms, do something like this:
WHERE my_gsi_pk = {pk value}
AND my_gsi_sk IN ({sk value 1}, {sk value 2})
I've looked in the Boto3 documentation in the .query() section and concentrated upon the KeyConditionExpression syntax but can't identify whether this is possible or not.
The query API does not support the IN operator in the KeyConditionExpression.
Use the execute_statement API instead. This executes a PartiQL statement, which does accept the IN operator in query operations for the Partition and Sort keys:
sk = ["Foo", "Bar"]
res = client.execute_statement(
Statement=f'SELECT * FROM "my_table"."my_gsi" WHERE my_gsi_pk = ? AND my_gsi_sk IN [{",".join(["?" for k in sk])}]',
Parameters= [{"S": "1"}] + [{"S": k} for k in sk]
)
This creates a PartiQL Statement like SELECT * FROM "my_table"."my_gsi" WHERE my_gsi_pk = ? AND my_gsi_sk IN [?, ?] and substitution Parameters like [{"S": "1"}, {"S": "Foo"}, {"S": "Bar"}].
Please note that the PartiQL will spend much more RCU than the Query. You can check this by requesting ReturnConsumedCapacity = ReturnConsumedCapacity.TOTAL
This is my first time using boto3 to query items from my DynamoDB and I can't figure out how to grab a certain value.
My table has a primary key of "Company" and a sort key of "DailyPrice".
I looked at the boto3 docs and used the example they had and I'm able to return all of the information related to AAPL by searching for that key value.
Here's my python script
import boto3
client = boto3.client('dynamodb')
response = client.query(
ExpressionAttributeValues={
':AAPL': {
'S': 'AAPL',
},
},
KeyConditionExpression='Company = :AAPL',
TableName='stock_tracker',
)
number_of_days = response['Count']
items = response['Items']
print(items)
Here's the response
{'Items': [
{'Company': {'S': 'AAPL'}, 'DailyPrice': {'S': '142.56'}},
{'Company': {'S': 'AAPL'}, 'DailyPrice': {'S': '154.51'}},
{'Company': {'S': 'AAPL'}, 'DailyPrice': {'S': '156.77'}}],
'Count': 3,
'ScannedCount': 3,}
I basically want to grab the daily price of every item for AAPL, because I want to add them all up in a separate python script. I'm not sure how I can grab the daily price specifically using my DynamoDB query
Your life will be easier with boto3.resource than boto3.client because you don't need all that 'S' type stuff around.
Here's a repo with sample code:
https://github.com/aws-samples/aws-dynamodb-examples/blob/master/DynamoDB-SDK-Examples/python/WorkingWithQueries/query_equals.py
Then just loop over the returned values in Python.
I have MariaDB 10.1. - I can't use JSON functions - JSON_EXTRACT etc.).
In the database I have a table CONTRACTS and a column data, which contains JSON (data type TEXT):
{"879": "Test", "880": "15255", "881": "2021-10-22"}
And I need to find all records that have a key value of "880" in some range, eg greater than 10000 and less than 20000, ie. in this case, a record with a value of 15255.
Thanks for advice.
Maybe something like this:
SELECT
TRIM(BOTH '"' FROM
REGEXP_SUBSTR(REGEXP_SUBSTR(CONTRACTS.`data`, '"880": "[0-9]+"'), '"[0-9]+"$')
) * 1 BETWEEN 10000 AND 20000
FROM
(SELECT
'{"879": "Test", "880": "15255", "881": "2021-10-22"}' AS `data`
) AS CONTRACTS
So the most internal regexp gives you the key + value. The outer regexp takes that result and extracts the value in quotes. Trim the quotes and test the value. You could use the entire TRIM(...) as a criterium .
I'm trying to make a python3 sqlite3 database function that determines the amount of column with a for loop. During this for loop I want to add a "?" to a variable for each column.
So what I'm trying to do now is create a list from which I remove the quotes and brackets afterwards but using the .strip() and .replace() method don't seem to work for me either.
lijst = []
lijst.append("?")
lijst.append("?")
lijst.replace('"', '')
lijst.strip([])
print(lijst)
In the end I want it to look a bit like this (?, ?, ?).
lijst is a list, not a string. You can't treat the two interchangeably. Instead, you can construct a string directly. Where n in the number of ? you want in your final string:
n = 3
result = "({})".format(", ".join("?"*n))
print(result)
# (?, ?, ?)