I'm currently experimenting with Airflow for monitoring tasks regarding Snowflake and I'd like to execute a simple DAG with one task that pushes a SQL query to in Snowflake and should check the returned value that should be a number to be greater than a defined threshold.
So the following is basically my sql Statement in the DAG definition:
query_check = """select COUNT(*)
FROM (select CASE WHEN NAME LIKE '%SW_PRODUCTFEED%' THEN 'PRODUCTFEED'
ELSE NULL END AS TASKTREE_NAME
, NAME
, STATE
, ERROR_MESSAGE
, SCHEDULED_TIME
, QUERY_START_TIME
, NEXT_SCHEDULED_TIME
from table(TEST_DB.INFORMATION_SCHEMA.task_history())
where TASKTREE_NAME IS NOT NULL
qualify DENSE_RANK() OVER (PARTITION BY TASKTREE_NAME ORDER BY to_date(SCHEDULED_TIME) desc) < 3
order by scheduled_time desc);"""
Then the following is the definition of the DAG and the task within it:
with dag:
query1_exec = SnowflakeCheckOperator(
task_id="snowflake_check_task_history",
sql=query_check,
params={
"check_name": "number_rows",
"check_statement": "count >=1"
},
conn_id="Snowflake_test"
)
query1_exec
I'd like to use the SnowflakeCheckOperator to check the returned value from the query if it's greater than 1
However, it seems that Snowflake or the SnowflakeOperator in that case is returning the result of the query in a dict object, like so:
Record: {'COUNT(*)': 10}
Therefore the check always results in a true statement because the SnowflakeCheckOperator isn't checking against the value of the Record["Count"] but something else.
Now my question is how to handle the return value so that the check is evaluated against right value? Is it possible to change the format of the return value? Or maybe get access to the value of the key of the dict object?
Related
I have a flask API and I'm using Flask-SQLAlchemy to handle a SQLite database. I have a table which stores log entries, and I want to limit the maximum number of rows to a number n. Since insertions are also made from another script outside of flask using raw SQL, I created a trigger that checks the number of rows and deletes the oldest ones if the number is higher than n:
CREATE TRIGGER 'trigger_log_insert'
BEFORE INSERT ON 'connection_logs'
WHEN ( SELECT count(*) FROM 'connection_logs' ) > 5
BEGIN
DELETE FROM 'connection_logs'
WHERE id NOT IN ( SELECT id FROM 'connection_logs' ORDER BY id DESC LIMIT 5 );
END
This trigger works as expected, but I am struggling to set it using flask-sqlalchemy. How can I set the trigger / execute raw SQL using flask-sqlalchemy? The SQL only needs to be executed once after db creation so I intent to execute it right after my create_all() statement.
I stumbled upon this StackOverflow answer which suggested a solution that is apparently going to be deprecated soon. I also read the SQLAlchemy documentation about custom DDL, but I don't know how to create this custom DDL with flask_sqlalchemy. When I create the DDL like in the SQLAlchemy documentation, I get an error message saying
DDL object is not bound to an Engine or Connection.:
trigger = DDL(
"CREATE TRIGGER 'trigger_log_insert'"
"BEFORE INSERT ON 'connection_logs'"
"WHEN ( SELECT count(*) FROM 'connection_logs' ) > 5"
"BEGIN"
"DELETE FROM 'connection_logs' WHERE id NOT IN"
"("
"SELECT id FROM 'connection_logs' ORDER BY id DESC LIMIT 5"
");"
"END"
)
event.listen(ConnectionLog, 'after_create', trigger.execute())
My model is defined using flask-sqlalchemy's declarative base model:
class ConnectionLog(db.Model):
__tablename__ = 'connection_logs'
You don't need to create a DDL instance, you can execute the SQL within the listener function. The relevant docs are here.
import sqlalchemy as sa
...
class ConnectionLog(db.Model):
__tablename__ = 'connection_logs'
...
def after_create(target, connection, **kw):
connection.execute(sa.text("""\
CREATE TRIGGER 'trigger_log_insert'
BEFORE INSERT ON 'connection_logs'
WHEN ( SELECT count(*) FROM 'connection_logs' ) > 5
BEGIN
DELETE FROM 'connection_logs' WHERE id NOT IN
(
SELECT id FROM 'connection_logs' ORDER BY id DESC LIMIT 5
);
END
"""
))
# Listen on the underlying table object, not on the model class.
sa.event.listen(ConnectionLog.__table__, "after_create", after_create)
Ensure that the interpreter has read this code before creating the tables.
I have set my column to int not null default 1... but whenever I save my record, it sets default value for that record to be 0.
I am not setting it anywhere. I don't know where I am making a mistake.
I have debugged my code , and when I am passing new entity object it is setting default value for not null to 0 .May be it is something with LINQ, But I don't know how to handle it.I don't want to explicitly assign value.
Thanks!
For sql-server, you can use SQL Server Profiler to catch all the scripts you run into the DB.
This may show you some details
Try running this query, replacing the 'myTable' and 'myColumn' values with your actual TABLE and COLUMN names, and see what's returned:
SELECT
OBJECT_NAME(C.object_id) AS [Table Name]
,C.Name AS [Column Name]
,DC.Name AS [Constraint Name]
,DC.Type_Desc AS [Constraint Type]
,DC.Definition AS [Default Value]
FROM sys.default_constraints DC
INNER JOIN sys.Columns C
ON DC.parent_column_id = C.column_id
AND DC.parent_object_id = C.object_id
WHERE OBJECT_NAME(DC.parent_object_id) = 'myTable'
AND COL_NAME(DC.parent_object_id,DC.parent_column_id) = 'myColumn'
;
Should return something like this:
[Table Name] [Column Name] [Constraint Name] [Constraint Type] [Default Value]
-------------------------------------------------------------------------------------------
myTable myColumn DF_myTable_myColumn DEFAULT_CONSTRAINT ('0')
If the [Default Value] returned is indeed (1), then it means that you have set the constraint properly and something else is at play here. It might be a trigger, or some other automated DML that you've forgotten/didn't know about, or something else entirely.
I am not the world's biggest fan of using a TRIGGER, but in a case like this, it could be handy. I find that one of the best uses for a TRIGGER is debugging little stuff like this - because it lets you see what values are being passed into a table without having to scroll through mountains of profiler data. You could try something like this (again, switching out the myTable and myColumn values with your actual table and column names):
CREATE TABLE Default_Check
(
Action_Time DATETIME NOT NULL DEFAULT GETDATE()
,Inserted_Value INT
);
CREATE TRIGGER Checking_Default ON myTable
AFTER INSERT, UPDATE
AS
BEGIN
INSERT INTO Default_Check (Inserted_Value)
SELECT I.myColumn
FROM Inserted I
;
END
;
This trigger would simply list the date/time of an update/insert done against your table, as well as the inserted value. After creating this, you could run a single INSERT statement, then check:
SELECT * FROM Default_Check;
If you see one row, only one action (insert/update) was done against the table. If you see two, something you don't expect is happening - you can check to see what. You will also see here when the 0 was inserted/updated.
When you're done, just make sure you DROP the trigger:
DROP TRIGGER Checking_Default;
You'll want to DROP the table, too, once it's become irrelevant:
DROP TABLE Default_Check;
If all of this still didn't help you, let me know.
In VB use
Property VariableName As Integer? = Nothing
And
In C# use
int? value = 0;
if (value == 0)
{
value = null;
}
Please check My Example:
create table emp ( ids int null, [DOJ] datetime NOT null)
ALTER TABLE [dbo].[Emp] ADD CONSTRAINT DF_Emp_DOJ DEFAULT (GETDATE()) FOR [DOJ]
1--Not working for Default Values
insert into emp
select '1',''
2 ---working for Default Values
insert into emp(ids) Values(13)
select * From emp
My code follows:
SELECT COUNT(_id) AS count FROM general WHERE _id = 1 CASE WHEN count > 0 THEN UPDATE general SET userGivenId = 'xxx' WHERE _id = 1 ELSE INSERT INTO general (userGivenId) VALUES ('xxx' ) END
With the error:
android.database.sqlite.SQLiteException: near "CASE": syntax error: , while compiling: SELECT COUNT(_id) AS count FROM general WHERE _id = 1 CASE WHEN count > 0 THEN UPDATE general SET userGivenId = 'xxx' WHERE _id = 1 ELSE INSERT INTO general (userGivenId) VALUES ('xxx' ) END
This is the shortest query I will use. Why I do this is because my other queries will have rows that needs to be updated but some may not be touched. Using replace will replace all the data (at least that is how it works for me on my Android phone). For instance my File class will have a filePath, but sometimes the response from the server will return null and I am only to update the File IF the server returns a new File.
Did I forget to write anything?
SQLite does not have any control logic because this would not make sense in an embedded database (there is no separate server machine/process whose asynchronous execution could improve performance).
CASE can be used only for expressions, not for entire commands.
Handle the control logic in your app:
Cursor c = db.rawQuery("SELECT 1 FROM general WHERE _id = 1", null);
if (c.moveToFirst())
db.execSQL("UPDATE general SET userGivenId = 'xxx' WHERE _id = 1");
else
db.execSQL("INSERT INTO general (userGivenId) VALUES ('xxx')");
For these particular commands, if you have a unique constraint on the _id column (and a primary key is constrained to be unique), you can use the INSERT OR REPLACE command:
INSERT OR REPLACE INTO general(_id, userGivenId) VALUES(1, 'xxx')
I am working on Oracle 11g Db, Having trouble on writing Oracle syntax.
I am trying to pass a number variable to my select query and populate the select query to a cursor.
Declare yr_nr NUMBER;
Begin
yr_nr := 2014;
SELECT DCD.CCY ID, DCD.CCYCDDSC DSC
FROM CCYDCD DCD, CCYEXC EXC
WHERE DCD.CCY = EXC.CCY
AND EXC.YEARNR = yr_nr
End
This select query returns 80 records. How to rewrite this syntax.
Ok, so what you have here is an anonymous block and everything that happens in the block stays in that block. Kinda like Vegas.
In other words there is nothing to handle the result set from your query. When you do this:
declare
[varName] [type]
begin
select foo from bar where column = var ; <--- this has no place to go!
end
When you are at an sqlPlus prompt, sqlPlus has a default record set handler which then processes the returned record set and prints it to the screen.
When you use any third party tool like JDBC or Oracle's own OCI library those provide a record set handler then parse them to you with the appropriate calls to get the data, e.g.:
rs.getInteger([query],[column] ) //which returns the specific value.
That anonymous block is essentially a stored procedure. So you have to have something to do with the result set. This is the cause of the missing "into" error you are getting.
If on the other hand you did something like:
declare
[varName] [type]
result number ;
begin
select count(foo) into result from bar where column = var ;
end
The variable result would have the value of 80 since that is the number of records fetched.
declare
[varName] [type]
cursor thisCursor(p1 in number ) is select foo from bar where column = p1 ;
begin
for rec in thisCursor(varName) loop
If rec.column = [some value] then
doSomething
end if ;
end loop ;
end
Do this would allow you to do something with the result set.
I'm using the SQLite C interface to write an application. Since I like security, I'm using prepared statements to query the database. In one such query, I'm selecting rows from a virtual database using the MATCH keyword for full-text-searching. Here's an example:
SELECT * FROM Emails
WHERE ( Subject LIKE ?001 OR ?001 IS NULL )
AND ( Author LIKE ?002 OR ?002 IS NULL )
AND ( Body MATCH ?003 OR ?003 IS NULL )
This allows the user to enter any terms (Subject, Author, or Body) individually or in any combination to do a search. Any term that isn't entered, I'll bind NULL to that parameter. The problem with that statement is that you can't use the OR keyword with the MATCH keyword. I'm looking for a statement I can use with the MATCH keyword to return all rows if not searching in the Body column. Is there such a statement?
I suggest the following:
SELECT * FROM emails
WHERE ...
AND ( CASE (SELECT COUNT(*) FROM emails WHERE body MATCH ?003)
WHEN 0 THEN 1
ELSE body MATCH ?003
END )
I ended up modifying the SQL statement at runtime to replace MATCH with LIKE '%'. Not very elegant, but it works for now.