I am trying to build some smart error messages using the #app.errorhandler (500) feature. For example, my route includes an INSERT command to the database:
if request.method == "POST":
userID = int(request.form.get("userID"))
topicID = int(request.form.get("topicID"))
db.execute("BEGIN TRANSACTION")
db.execute("INSERT INTO UserToTopic (userID,topicID) VALUES (?,?)", userID, topicID)
db.execute("COMMIT")
If that transaction violates a constraint, such as UNIQUE or FOREIGN_KEY, I want to catch the error and display a user-friendly message. To do this, I'm using the Flask #app.errorhandler as follows:
#app.errorhandler(500)
def internal_error(error):
db.execute("ROLLBACK")
return render_template('500.html'), 500
The "ROLLBACK" command works fine if I'm in the middle of a database transaction. But sometimes the 500 error is not related to the db, and in those cases the ROLLBACK statement itself causes an error, because you can't rollback a transaction that never started. So I'm looking for a method that returns a Boolean value that would be true if a db transaction is under way, and false if not, so I can use it to make the ROLLBACK conditional. The only one I can find in the SQLite3 documentation is for a C interface, and I can't get it to work with my Python code. Any suggestions?
I know that if I'm careful enough with my forms and routes, I can prevent 99% of potential violations of db rules. But I would still like a smart error catcher to protect me for the other 1%.
I don't know how transaction works in sqlite but what you are trying to do, you can achieve it by try/except statements
use try/except within the function
try:
db.execute("ROLLBACK")
except:
pass
return render_template('500.html'), 500
Use try/except when inserting data.
from flask import abort
try:
userID = int(request.form.get("userID"))
[...]
except:
db.rollback()
abort(500)
I am not familiar with sqlite errors, if you know what specific error occurs except for that specific error.
Related
I am using this query to get some statistics from our DB
SELECT DateTimePart("yyyy", c.RedeemedDate) AS RedeemedInYear,
DateTimePart("m", c.RedeemedDate) AS RedeemedInMonth,
AVG(c.RedPriceStoreSummary ?? c.WhitePriceStoreSummary) AS AverageBasketValue
FROM c
WHERE c.IsRedeemed = true
AND c.Brand = 'xxx'
AND c.RedeemedDate != null
AND DateTimePart("yyyy", c.RedeemedDate) = DateTimePart("yyyy", GetCurrentDateTime())
GROUP BY DateTimePart("m", c.RedeemedDate),
DateTimePart("yyyy", c.RedeemedDate)
The problem is that query crashes with the following error if there are no results
Cannot read property 'create' of undefined
If I force the query to where it gets results then everything works fine but I dont want the query to crash if there are no results I want an empty result set.
Am I missing something here?
You need to trap queries in a try block and catch if it returns a 404. This is a hold over in behavior from the previous SDK. Alternatively you can use the stream variant of these SDK functions which will not throw exceptions when data is not found, however you still want to check the http response code and you'll also need to manually deserialize the data.
I use airflow python operators to execute sql queries against a redshift/postgres database. In order to debug, I'd like the DAG to return the results of the sql execution, similar to what you would see if executing locally in a console:
I'm using psycop2 to create a connection/cursor and execute the sql. Having this logged would be extremely helpful to confirm the parsed parameterized sql, and confirm that data was actually inserted (I have painfully experiences issues where differences in environments caused unexpected behavior)
I do not have deep knowledge of airflow or the low level workings of the python DBAPI, but the pscyopg2 documentation does seem to refer to some methods and connection configurations that may allow this.
I find it very perplexing that this is difficult to do, as I'd imagine it would be a primary use case of running ETLs on this platform. I've heard suggestions to simply create additional tasks that query the table before and after, but this seems clunky and ineffective.
Could anyone please explain how this may be possible, and if not, explain why? Alternate methods of achieving similar results welcome. Thanks!
So far I have tried the connection.status_message() method, but it only seems to return the first line of the sql and not the results. I have also attempted to create a logging cursor, which produces the sql, but not the console results
import logging
import psycopg2 as pg
from psycopg2.extras import LoggingConnection
conn = pg.connect(
connection_factory=LoggingConnection,
...
)
conn.autocommit = True
logging.basicConfig(level=logging.DEBUG)
logger = logging.getLogger(__name__)
logger.addHandler(logging.StreamHandler(sys.stdout))
conn.initialize(logger)
cur = conn.cursor()
sql = """
INSERT INTO mytable (
SELECT *
FROM other_table
);
"""
cur.execute(sql)
I'd like the logger to return something like:
sql> INSERT INTO mytable (
SELECT ...
[2019-07-25 23:00:54] 912 rows affected in 4 s 442 ms
Let's assume you are writing an operator that uses postgres hook to do something in sql.
Anything printed inside an operator is logged.
So, if you want to log the statement, just print the statement in your operator.
print(sql)
If you want to log the result, fetch the result and print the result.
E.g.
result = cur.fetchall()
for row in result:
print(row)
Alternatively you can use self.log.info in place of print, where self refers to the operator instance.
Ok, so after some trial and error I've found a method that works for my setup and objective. To recap, my goal is to run ETL's via python scripts, orchestrated in Airflow. Referring to the documentation for statusmessage:
Read-only attribute containing the message returned by the last command:
The key is to manage logging in context with transactions executed on the server. In order for me to do this, I had to specifically set con.autocommit = False, and wrap SQL blocks with BEGIN TRANSACTION; and END TRANSACTION;. If you insert cur.statusmessage directly following a statement that deletes or inserts, you will get a response such as 'INSERT 0 92380'.
This still isn't as verbose as I would prefer, but it is a much better than nothing, and is very useful for troubleshooting ETL issues within Airflow logs.
Side notes:
- When autocommit is set to False, you must explicitly commit transactions.
- It may not be necessary to state transaction begin/end in your SQL. It may depend on your DB version.
con = psy.connect(...)
con.autocommit = False
cur = con.cursor()
try:
cur.execute([some_sql])
logging.info(f"Cursor statusmessage: {cur.statusmessage})
except:
con.rollback()
finally:
con.close()
There is some buried functionality within psycopg2 that I'm sure can be utilized, but the documentation is pretty thin and there are no clear examples. If anyone has suggestions on how to utilize things such as logobjects, or returning join PID to somehow retrieve additional information.
I am migrating my app's SQLite helper to Room. Basically what I am doing is just copying data from old SQLite database to Room, so due to schema mismatch I need to provide migration. I am having this issue with BLOB data in Room.
I have below simple model
class NewCourse {
var weekday: Array<String> = arrayOf()
}
I also have TypeConverter as
#TypeConverter
fun toArray(concatenatedStrings: String?): Array<String>? {
return concatenatedStrings?.split(",".toRegex())?.dropLastWhile { it.isEmpty() }?.toTypedArray()
}
#TypeConverter
fun fromArray(strings: Array<String>?): String? {
return strings?.joinToString(",")
}
Within my old appdatabase.db database I have a corresponding table Course with a field weekday which has a type BLOB.
Well, because of my TypeConverter in room database I will have weekday with a type TEXT. While migrating I running below SQL script.
INSERT INTO NewCourse (weekday) SELECT weekday FROM Course
As weekday from Course table is BLOB type and in SQL you can basically store anything to anything, am I expecting it will copy BLOB-typed weekday in Course to TEXT-typed weekday in NewCourse.
Well at first I was expecting some error due to type mismatch. But "fortunately", but not expectedly, Room doesn't throw any exception and it gets value of BLOB and copies as TEXT.
My first question was why it is working? i.e. how it's copying the TEXT value of BLOB to my newly created table?
I never cared about it, as it was working perfectly, until I did some testing with Robolectic.
Unfortunately, I am getting error if I start testing with Robolectic. After copying data in my migration, when I query for NewCourse I am getting SQL error of
android.database.sqlite.SQLiteException: Getting string when column is blob. Row 0, col 10
So, I suppose here it is copying the data as BLOB and when querying for weekday it is throwing an exception as getWeekDay calls getString of cursor.
My second question would be "Why while testing with Robolectic it is not working as it is working with just running the app?"
I also tested the queries with just Sql not involving Android, and over there it copies the BLOB as BLOB even though the type of weekday at NewCourse is TEXT as expected.
Robolectric is a testing library for android applications. The keyword here is testing and by that it means there shouldn't be any exception. Robolectric showing you error maybe because of some android devices may throw exception so your application will crash. Try checking your logs while your application running. Maybe you are missing some warnings.
We are using the tSQLt framework and have the below code in the test.
IF #count>0
EXEC tsqlt.fail;
else EXEC tSQLt.AssertEquals 1,1;
I am interested to know how we can display a custom test success or failure message when this test gets executed?
tSQLt.fail takes up to 10 parameters that all get concatenated into a custom failure message.
You also do not need the call to tSQLt.AssertEquals as it, in your case, literally does nothing.
BTW, asserting a count is in almost all cases a bad idea, as it does not really tell you anything about the result. If you get the correct count back, you could still have wrong data. And if you get the incorrect count, you don't have any additional info on what went wrong.
Have a look at tSQLt.AssertEqualsTable or tSQLt.AssertEmptyTable instead.
When starting my program for the first time since the associated table has been deleted I get this error:
An exception of type 'Microsoft.WindowsAzure.Storage.StorageException' occurred in Microsoft.WindowsAzure.Storage.dll but was not handled in user code
Additional information: The remote server returned an error: (409) Conflict.
However if I refresh the crashed page the table will successfully create.
Here is the code just in case:
CloudStorageAccount storageAccount = CloudStorageAccount.Parse(
Microsoft.WindowsAzure.CloudConfigurationManager.
GetSetting("StorageConnectionString"));
CloudTableClient tableClient = storageAccount.CreateCloudTableClient();
CloudTable table = tableClient.GetTableReference("tableTesting");
table.CreateIfNotExists();
I don't really understand how or why I'd be getting a conflict error if there's nothing there.
These errors appear elsewhere in my code as well when I'm working with blob containers, but I can't reproduce them as easily.
If you look at the status codes here: http://msdn.microsoft.com/en-us/library/azure/dd179438.aspx, you will notice that you get 409 error code in two scenarios:
Table already exists
Table is being deleted
If I understand correctly, table.CreateIfNotExists() only handles the 1st situation but not the 2nd one. Please check if that is not the case in your situation. One way to check this would be to see details of Storage Exception. Somewhere you should get the code which would match with the link I mentioned above.
Also one important thing to understand is that when you delete the table, it is actually marked for deletion and is actually deleted through a background process (much like garbage collection). If you try to create a table between these two steps, you will get the 2nd error.