How to know which foreign key constraint failed in SQLite? [duplicate] - sqlite

I get the SQLite error message "FOREIGN KEY constraint failed". That's the complete error information (besides a part of the SQL query) and it's not helpful. (In fact it's just as good (or bad) as Oracle error messages.) I need to know the name of the constraint to investigate the issue in my program. Unfortunately there's no web support platform to discuss this with an SQLite community. Does somebody know how to get more information about the error out of that SQLite library?
I'm specifically using the System.Data.SQLite library for .NET but the error message comes directly from the core and there are no additional exception properties that could help me.

Due to the way in which deferred FK constraints are implemented in SQLite, this information is not available when the error is raised.
You could reimplement the FK checks as triggers.
Alternatively, log the values in the failed command, and look up the data by hand.

In a Django project, here below is what I did to replace FOREIGN KEY constraint failed with (for example) DETAIL: Key (base_id)=(1a389dc3-5bc1-4132-8a4a-c8200533503a) is not present in table "backend_base" ...
The solution is Django specific since it's based on Django ORM's methods; obj is supposed to be the Model instance which caused the exception upon model.save()
from django.core.exceptions import ObjectDoesNotExist
def explain_integrity_errors(obj):
"""
Replace 'FOREIGN KEY constraint failed' error message provided by sqlite
with something useful (i.e. the exception thrown by PostgreSQL):
'DETAIL: Key (base_id)=(1a389dc3-5bc1-4132-8a4a-c8200533503a) is not present in table "backend_base"'
"""
error_msg = ''
errors = []
# Scan all FKs
relation_fields = [f for f in obj._meta.concrete_fields if f.is_relation]
for field in relation_fields:
try:
# try to access the related target
fk_target = getattr(obj, field.name)
except ObjectDoesNotExist as e:
# Log offending FK
fk_field = field.name + '_id'
fk_value = getattr(obj, fk_field)
fk_table = field.related_model._meta.db_table
errors.append('Key (%s)=(%s) is not present in table "%s"' % (
fk_field,
str(fk_value),
fk_table
))
if len(errors):
error_msg = ' DETAIL: ' + "; ".join(errors)
return error_msg

Related

How can my Flask app check whether a SQLite3 transaction is in progress?

I am trying to build some smart error messages using the #app.errorhandler (500) feature. For example, my route includes an INSERT command to the database:
if request.method == "POST":
userID = int(request.form.get("userID"))
topicID = int(request.form.get("topicID"))
db.execute("BEGIN TRANSACTION")
db.execute("INSERT INTO UserToTopic (userID,topicID) VALUES (?,?)", userID, topicID)
db.execute("COMMIT")
If that transaction violates a constraint, such as UNIQUE or FOREIGN_KEY, I want to catch the error and display a user-friendly message. To do this, I'm using the Flask #app.errorhandler as follows:
#app.errorhandler(500)
def internal_error(error):
db.execute("ROLLBACK")
return render_template('500.html'), 500
The "ROLLBACK" command works fine if I'm in the middle of a database transaction. But sometimes the 500 error is not related to the db, and in those cases the ROLLBACK statement itself causes an error, because you can't rollback a transaction that never started. So I'm looking for a method that returns a Boolean value that would be true if a db transaction is under way, and false if not, so I can use it to make the ROLLBACK conditional. The only one I can find in the SQLite3 documentation is for a C interface, and I can't get it to work with my Python code. Any suggestions?
I know that if I'm careful enough with my forms and routes, I can prevent 99% of potential violations of db rules. But I would still like a smart error catcher to protect me for the other 1%.
I don't know how transaction works in sqlite but what you are trying to do, you can achieve it by try/except statements
use try/except within the function
try:
db.execute("ROLLBACK")
except:
pass
return render_template('500.html'), 500
Use try/except when inserting data.
from flask import abort
try:
userID = int(request.form.get("userID"))
[...]
except:
db.rollback()
abort(500)
I am not familiar with sqlite errors, if you know what specific error occurs except for that specific error.

How can I log sql execution results in airflow?

I use airflow python operators to execute sql queries against a redshift/postgres database. In order to debug, I'd like the DAG to return the results of the sql execution, similar to what you would see if executing locally in a console:
I'm using psycop2 to create a connection/cursor and execute the sql. Having this logged would be extremely helpful to confirm the parsed parameterized sql, and confirm that data was actually inserted (I have painfully experiences issues where differences in environments caused unexpected behavior)
I do not have deep knowledge of airflow or the low level workings of the python DBAPI, but the pscyopg2 documentation does seem to refer to some methods and connection configurations that may allow this.
I find it very perplexing that this is difficult to do, as I'd imagine it would be a primary use case of running ETLs on this platform. I've heard suggestions to simply create additional tasks that query the table before and after, but this seems clunky and ineffective.
Could anyone please explain how this may be possible, and if not, explain why? Alternate methods of achieving similar results welcome. Thanks!
So far I have tried the connection.status_message() method, but it only seems to return the first line of the sql and not the results. I have also attempted to create a logging cursor, which produces the sql, but not the console results
import logging
import psycopg2 as pg
from psycopg2.extras import LoggingConnection
conn = pg.connect(
connection_factory=LoggingConnection,
...
)
conn.autocommit = True
logging.basicConfig(level=logging.DEBUG)
logger = logging.getLogger(__name__)
logger.addHandler(logging.StreamHandler(sys.stdout))
conn.initialize(logger)
cur = conn.cursor()
sql = """
INSERT INTO mytable (
SELECT *
FROM other_table
);
"""
cur.execute(sql)
I'd like the logger to return something like:
sql> INSERT INTO mytable (
SELECT ...
[2019-07-25 23:00:54] 912 rows affected in 4 s 442 ms
Let's assume you are writing an operator that uses postgres hook to do something in sql.
Anything printed inside an operator is logged.
So, if you want to log the statement, just print the statement in your operator.
print(sql)
If you want to log the result, fetch the result and print the result.
E.g.
result = cur.fetchall()
for row in result:
print(row)
Alternatively you can use self.log.info in place of print, where self refers to the operator instance.
Ok, so after some trial and error I've found a method that works for my setup and objective. To recap, my goal is to run ETL's via python scripts, orchestrated in Airflow. Referring to the documentation for statusmessage:
Read-only attribute containing the message returned by the last command:
The key is to manage logging in context with transactions executed on the server. In order for me to do this, I had to specifically set con.autocommit = False, and wrap SQL blocks with BEGIN TRANSACTION; and END TRANSACTION;. If you insert cur.statusmessage directly following a statement that deletes or inserts, you will get a response such as 'INSERT 0 92380'.
This still isn't as verbose as I would prefer, but it is a much better than nothing, and is very useful for troubleshooting ETL issues within Airflow logs.
Side notes:
- When autocommit is set to False, you must explicitly commit transactions.
- It may not be necessary to state transaction begin/end in your SQL. It may depend on your DB version.
con = psy.connect(...)
con.autocommit = False
cur = con.cursor()
try:
cur.execute([some_sql])
logging.info(f"Cursor statusmessage: {cur.statusmessage})
except:
con.rollback()
finally:
con.close()
There is some buried functionality within psycopg2 that I'm sure can be utilized, but the documentation is pretty thin and there are no clear examples. If anyone has suggestions on how to utilize things such as logobjects, or returning join PID to somehow retrieve additional information.

Room unexpected behavior when working with BLOB & TEXT

I am migrating my app's SQLite helper to Room. Basically what I am doing is just copying data from old SQLite database to Room, so due to schema mismatch I need to provide migration. I am having this issue with BLOB data in Room.
I have below simple model
class NewCourse {
var weekday: Array<String> = arrayOf()
}
I also have TypeConverter as
#TypeConverter
fun toArray(concatenatedStrings: String?): Array<String>? {
return concatenatedStrings?.split(",".toRegex())?.dropLastWhile { it.isEmpty() }?.toTypedArray()
}
#TypeConverter
fun fromArray(strings: Array<String>?): String? {
return strings?.joinToString(",")
}
Within my old appdatabase.db database I have a corresponding table Course with a field weekday which has a type BLOB.
Well, because of my TypeConverter in room database I will have weekday with a type TEXT. While migrating I running below SQL script.
INSERT INTO NewCourse (weekday) SELECT weekday FROM Course
As weekday from Course table is BLOB type and in SQL you can basically store anything to anything, am I expecting it will copy BLOB-typed weekday in Course to TEXT-typed weekday in NewCourse.
Well at first I was expecting some error due to type mismatch. But "fortunately", but not expectedly, Room doesn't throw any exception and it gets value of BLOB and copies as TEXT.
My first question was why it is working? i.e. how it's copying the TEXT value of BLOB to my newly created table?
I never cared about it, as it was working perfectly, until I did some testing with Robolectic.
Unfortunately, I am getting error if I start testing with Robolectic. After copying data in my migration, when I query for NewCourse I am getting SQL error of
android.database.sqlite.SQLiteException: Getting string when column is blob. Row 0, col 10
So, I suppose here it is copying the data as BLOB and when querying for weekday it is throwing an exception as getWeekDay calls getString of cursor.
My second question would be "Why while testing with Robolectic it is not working as it is working with just running the app?"
I also tested the queries with just Sql not involving Android, and over there it copies the BLOB as BLOB even though the type of weekday at NewCourse is TEXT as expected.
Robolectric is a testing library for android applications. The keyword here is testing and by that it means there shouldn't be any exception. Robolectric showing you error maybe because of some android devices may throw exception so your application will crash. Try checking your logs while your application running. Maybe you are missing some warnings.

Perl dancer error execute method undefined

I was trying to make a website using perl dancer, below is my code. It seems to be correct but the page keeps loading and never enters the values in the database. When I cancel the page I get an error stating "request to POST /appform crashed: Can't call method "execute" on an undefined value". I can't figured out whats wrong in the code. If you have any other code please mention.
I am using SQLite for database.
There is a database campus.dband I am inserting the value in student table.
post '/appform' => sub {
my $q = CGI ->new;
my $name = $q->param ("firstname");
my $password = $q->param("password");
my $mobile_no = $q->param("mobile");
my $gender = $q->param("gender");
my $email = $q->param("email");
my $address = $q->param("address");
my $sslc = $q->param("SSLC");
my $hsc = $q->param("HSC");
my $cgpa = $q->param("cgpa");
my $languages = $q->param("lang");
my $internships = $q->param("intern");
my $preferred_loc = $q->param("country");
my $sql = "insert into student(name,mobile_no,gender,email,address,sslc,hsc,cgpa,languages,internships,preferred_loc,password,applied_job,company_applied) values ('?','?','?','?','?','?','?','?','?','?','?','?','?','?');";
my $sth = database->prepare($sql);
$sth->execute($name,$mobile_no,$gender,$email,$address,$sslc,$hsc,$cgpa,$languages,$internships,$preferred_loc,$password) or die $sth->errstr;
#$sth->execute();
$sth-> finish;
set_flash('New entry posted!');
redirect '/';
};
You're using the database keyword to get a database handle. I'm guessing that's coming from Dancer2::Plugin::Database (it would be useful if you could include information like this in your question).
The error says that you're calling execute() on an undefined value. You're calling execute() on the variable $sth. So $sth is undefined. You get $sth by calling prepare() on the database handle returned from database(). So it looks like the prepare() call is failing. You should check the return value from that call and throw an error if it fails.
The most common reason for prepare() to fail is that you're trying to compile an SQL statement that contains an error. I can't see any obvious error in your SQL, but it's worth checking it by running it manually against your database.
I see you're using bind params in your SQL statement. That's a great idea, but please note that you don't need to quote the question marks in your SQL - the database driver will handle that for you. I don't think that's what is causing your problem though.
I also see that you're using CGI.pm inside your Dancer app to get the request parameters. To be honest, I'm slightly surprised that it works - but it's a terrible idea. Dancer has its own keywords that will give you this information. Look at query_parameters(), body_parameters() and route_parameters() in the Dancer documentation.
In addition to the points made already, that your DBI prepare() call is probably failing (add error-checking to see why, e.g. my $sth = database->prepare('...') or die "DB error: " . database->errstr) and that you're using CGI.pm within a Dancer app (... don't do that, I'm surprised it would work at all - look at the Dancer documentation for how to access the params your app was sent), look also at the quick_insert convenience method provided by Dancer::Plugin::Database / Dancer2::Plugin::Database so that you don't have to write that SQL INSERT statement at all.

AmazonServiceException: Supplied AttributeValue is empty, must contain exactly one of the supported datatypes

I am trying to import data from dynamodb console interface, but unable to get success.
Data is
{"_id":{"s":"d9922db0-83ac-11e6-9263-cd3ebf92dec3"},"applicationId":{"S":"2"},"applicationName":{"S":"Paperclip"},"ip":{"S":"127.0.0.1"},"objectInfo":{"S":"elearning_2699"},"referalUrl":{"S":"backported data"},"url":{"S":""},"userAgent":{"S":""},"userEmail":{"S":"karthick.shivanna#test.com"},"userId":{"S":"508521"},"userName":{"S":"Karthik"},"created":{"S":"1486983137000"},"verb":{"S":"submitproject"},"dataVals":{"S":"{\"projectid\":5,\"name\":\"Test 1\",\"domain\":\"apparel\",\"submittype\":[\"Writeup\",\"Screenshots\"],\"passcriteria\":\"Percentage\",\"taemail\":\"bhargava.gade#test.com\",\"attemptNo\":1,\"submitDate\":1467784988}"},"eventTime":{"S":"1467784988000"}}
I am getting below error
Error: java.lang.RuntimeException:
com.amazonaws.AmazonServiceException: Supplied AttributeValue is
empty, must contain exactly one of the supported datatypes (Service:
AmazonDynamoDBv2; Status Code: 400; Error Code: ValidationException;
Request ID: GECS2L57CG9ANLKCSJSB8EIKVRVV4KQNSO5AEMVJF66Q9ASUAAJG) at
org.apache.hadoop.dynamodb.DynamoDBFibonacciRetryer.handleException(DynamoDBFibonacciRetryer.java:107)
at
org.apache.hadoop.dynamodb.DynamoDBFibonacciRetryer.runWithRetry(DynamoDBFibonacciRetryer.java:83)
at
org.apache.hadoop.dynamodb.DynamoDBClient.writeBatch(DynamoDBClient.java:220)
at
org.apache.hadoop.dynamodb.DynamoDBClient.putBatch(DynamoDBClient.java:170)
at
org.apache.hadoop.dynamodb.write.AbstractDynamoDBRecordWriter.write(AbstractDynamoDBRecordWriter.java:91)
at
org.apache.hadoop.mapred.MapTask$DirectMapOutputCollector.collect(MapTask.java:844)
at
org.apache.hadoop.mapred.MapTask$OldOutputCollector.collect(MapTask.java:596)
at org.apache.hadoop.dynamodb.tools.ImportMapper.map(ImportMapper.j
errorStackTrace
amazonaws.datapipeline.taskrunner.TaskExecutionException: Failed to
complete EMR transform. at
amazonaws.datapipeline.activity.EmrActivity.runActivity(EmrActivity.java:67)
at
amazonaws.datapipeline.objects.AbstractActivity.run(AbstractActivity.java:16)
at
amazonaws.datapipeline.taskrunner.TaskPoller.executeRemoteRunner(TaskPoller.java:136)
at
amazonaws.datapipeline.taskrunner.TaskPoller.executeTask(TaskPoller.java:105)
at
amazonaws.datapipeline.taskrunner.TaskPoller$1.run(TaskPoller.java:81)
at
private.com.amazonaws.services.datapipeline.poller.PollWorker.executeWork(PollWorker.java:76)
at
private.com.amazonaws.services.datapipeline.poller.PollWorker.run(PollWorker.java:53)
at java.lang.Thread.run(Thread.java:745) Caused by:
amazonaws.datapipeline.taskrunner.TaskExecutionException: Error:
java.lang.RuntimeException: com.amazonaws.AmazonServiceException:
Supplied AttributeValue is empty, must contain exactly one of the
supported datatypes (Service: AmazonDynamoDBv2; Status Code: 400;
Error Code: ValidationException; Request ID:
GECS2L57CG9ANLKCSJSB8EIKVRVV4KQNSO5AEMVJF66Q9ASUAAJG) at
org.apache.hadoop.dynamodb.DynamoDBFibonacciRetryer.handleException(DynamoDBFibonacciRetryer.java:107)
at
org.apache.hadoop.dynamodb.DynamoDBFibonacciRetryer.runWithRetry(DynamoDBFibonacciRetryer.java:83)
at
org.apache.hadoop.dynamodb.DynamoDBClient.writeBatch(DynamoDBClient.java:220)
at
org.apache.hadoop.dynamodb.DynamoDBClient.putBatch(DynamoDBClient.java:170)
at
org.apache.hadoop.dynamodb.write.AbstractDynamoDBRecordWriter.write(AbstractDynamoDBRecordWriter.java:91)
at
org.apache.hadoop.mapred.MapTask$DirectMapOutputCollector.collect(MapTask.java:844)
at
org.apache.hadoop.mapred.MapTask$OldOutputCollector.collect(MapTask.java:596)
at
org.apache.hadoop.dynamodb.tools.ImportMapper.map(ImportMapper.java:26)
at
org.apache.hadoop.dynamodb.tools.ImportMapper.map(ImportMapper.java:13)
at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:65) at
org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:432) at
org.apache.hadoop.mapred.MapTask.run(MapTask.java:343) at
org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:175) at
java.security.AccessController.doPrivileged(Native Method) at
javax.security.auth.Subject.doAs(Subject.java:415) at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:170) Caused
by: com.amazonaws.AmazonServiceException: Supplied AttributeValue is
empty, must contain exactly one of the supported datatypes (Service:
AmazonDynamoDBv2; Status Code: 400; Error Code: ValidationException;
Request ID: GECS2L57CG9ANLKCSJSB8EIKVRVV4KQNSO5AEMVJF66Q9ASUAAJG) at
com.amazonaws.http.AmazonHttpClient.handleErrorResponse(AmazonHttpClient.java:1182)
at
com.amazonaws.http.AmazonHttpClient.executeOneRequest(AmazonHttpClient.java:770)
at
com.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.java:489)
at
com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:310)
at
com.amazonaws.services.dynamodbv2.AmazonDynamoDBClient.invoke(AmazonDynamoDBClient.java:1772)
at
com.amazonaws.services.dynamodbv2.AmazonDynamoDBClient.batchWriteItem(AmazonDynamoDBClient.java:730)
at amazonaws.datapipeline.cluster.EmrUtil.runSteps(EmrUtil.java:286)
at
amazonaws.datapipeline.activity.EmrActivity.runActivity(EmrActivity.java:63)
Am I doing anything wrong?
Error: java.lang.RuntimeException: com.amazonaws.AmazonServiceException: Supplied AttributeValue is empty, must contain exactly one of the supported datatypes (Service: AmazonDynamoDBv2; Status Code: 400; Error Code: ValidationException This is the error you are getting.
Below are the possible reasons
DynamoDB does not support empty value, so you should remove those
fields (agree with #notionquest)
Field's value should have proper data type as per table
just updating here in case someone come across this again.Empty String and Binary attribute values are allowed
Attribute values of type String and Binary must have a length greater than zero if the attribute is used as a key attribute for a table or index.
I'm using Data pipeline with release label emr-5.23.0 and also encoutered the same problem. I solve it by using lower letter instead of capital letter for the Typing in the dynamo item. E.g instead of 'S' use 's', instead of 'N' use 'n'.
We have to go through step by step here. The above error occurred because the values for some of the attributes are empty. DynamoDB doesn't support empty value for the attributes.
Example: url, userAgent etc.
Please remove the empty attributes and try again. I can assure that the above issue will be resolved. However, something else could be wrong as well.
In my case , I got the same issue because of invalid parameter sends from
mapping template.
#set($inputRoot = $input.path('$'))
{
"userId": "$input.params('userId')",
"userEmail": "$input.params('userEmail')",
"userName": "$input.params('userName')",
"userPassword": "$input.params('userPassword')"
}
Here I sent extra parameter userId , that's why error occurred .
And for blank value , When you add an item, the primary key attribute(s) are the only required attributes. Attribute values cannot be null. String and Binary type attributes must have lengths greater than zero. Set type attributes cannot be empty. Requests with empty values will be rejected with a ValidationException exception.
Please check this document .
http://docs.aws.amazon.com/amazondynamodb/latest/APIReference/API_PutItem.html
I hope it will help you.
I had the same problem running a restore pipeline in aws. After looking for a while I found the problem. The AMI version of the restore was different from the export one.
I have other pipelines which work fine. I still don't know why in one case it didn't. Basically, I checked the AMI versions, they were 3.8.0 for the export one and 3.9.0 for the restore one. I change the restore to 3.8.0 and it works.
Here you will find a better explanation.
My two cents.
Use camel case, For example:
{"id":{"S":"123xyz"},"ip":{"S":"127.0.0.1"},"attempt":{"N":"10"},"allowed":{"BOOL":true}}
to
{"id":{"s":"123xyz"},"ip":{"s":"127.0.0.1"},"attempt":{"n":"10"},"allowed":{"bOOL":true}}
This is also one of the reasons for above error.

Resources