I am getting crazy. Now working a half day on my problem and can't find the solution/error. Creating a chat, I got a chat model and a chat_user model which has a user model as a foreign key.
Chat has has_many :chat_users, Test.Chat.ChatUser, foreign_key: :chat_id as has many, and the chat_user has belongs_to :user, Test.User and belongs_to :chat, Test.Chat.Chat.
I am preloading the associations with:
defp preload_associations(chat, params \\ nil) do
Repo.preload(chat, [
:chat_users
])
end
The Error i get is: ** (Postgrex.Error) ERROR 42703 (undefined_column): column c0.user_id does not exist
The table chat_users in the database has the field user_id with a foreign key. It seems to me, Ecto is searching for user_id in chat instead of chat_user.
Any idea what I am doing wrong? Thanks.
EDIT: Models
defmodule Test.Chat.ChatItem do
#moduledoc false
use Test.Web, :model
schema "chats" do
field :name, :string
...
has_many :chat_users, Test.Chat.ChatUser, foreign_key: :chat_id
timestamps()
end
defmodule Test.Chat.ChatUser do
use Test.Web, :model
schema "chat_users" do
...
belongs_to :user, Test.User
belongs_to :chat, Test.Chat.Chat
end
And
defmodule Test.User do
...
Finally I found the error. Had nothing to do with my migration or the models. The problem was caused by some environment configurations. The script, which builds the dev and test database did not drop the test database, so it had no user_id in the chat_users table. The error occured with mix test inside the docker container, so finding the error was not easy. I am new to elixir and still hate the stacktrace, used to read some nice c# stacktrace ;)
But thanks for the help.
Related
I am trying to build some smart error messages using the #app.errorhandler (500) feature. For example, my route includes an INSERT command to the database:
if request.method == "POST":
userID = int(request.form.get("userID"))
topicID = int(request.form.get("topicID"))
db.execute("BEGIN TRANSACTION")
db.execute("INSERT INTO UserToTopic (userID,topicID) VALUES (?,?)", userID, topicID)
db.execute("COMMIT")
If that transaction violates a constraint, such as UNIQUE or FOREIGN_KEY, I want to catch the error and display a user-friendly message. To do this, I'm using the Flask #app.errorhandler as follows:
#app.errorhandler(500)
def internal_error(error):
db.execute("ROLLBACK")
return render_template('500.html'), 500
The "ROLLBACK" command works fine if I'm in the middle of a database transaction. But sometimes the 500 error is not related to the db, and in those cases the ROLLBACK statement itself causes an error, because you can't rollback a transaction that never started. So I'm looking for a method that returns a Boolean value that would be true if a db transaction is under way, and false if not, so I can use it to make the ROLLBACK conditional. The only one I can find in the SQLite3 documentation is for a C interface, and I can't get it to work with my Python code. Any suggestions?
I know that if I'm careful enough with my forms and routes, I can prevent 99% of potential violations of db rules. But I would still like a smart error catcher to protect me for the other 1%.
I don't know how transaction works in sqlite but what you are trying to do, you can achieve it by try/except statements
use try/except within the function
try:
db.execute("ROLLBACK")
except:
pass
return render_template('500.html'), 500
Use try/except when inserting data.
from flask import abort
try:
userID = int(request.form.get("userID"))
[...]
except:
db.rollback()
abort(500)
I am not familiar with sqlite errors, if you know what specific error occurs except for that specific error.
I get the SQLite error message "FOREIGN KEY constraint failed". That's the complete error information (besides a part of the SQL query) and it's not helpful. (In fact it's just as good (or bad) as Oracle error messages.) I need to know the name of the constraint to investigate the issue in my program. Unfortunately there's no web support platform to discuss this with an SQLite community. Does somebody know how to get more information about the error out of that SQLite library?
I'm specifically using the System.Data.SQLite library for .NET but the error message comes directly from the core and there are no additional exception properties that could help me.
Due to the way in which deferred FK constraints are implemented in SQLite, this information is not available when the error is raised.
You could reimplement the FK checks as triggers.
Alternatively, log the values in the failed command, and look up the data by hand.
In a Django project, here below is what I did to replace FOREIGN KEY constraint failed with (for example) DETAIL: Key (base_id)=(1a389dc3-5bc1-4132-8a4a-c8200533503a) is not present in table "backend_base" ...
The solution is Django specific since it's based on Django ORM's methods; obj is supposed to be the Model instance which caused the exception upon model.save()
from django.core.exceptions import ObjectDoesNotExist
def explain_integrity_errors(obj):
"""
Replace 'FOREIGN KEY constraint failed' error message provided by sqlite
with something useful (i.e. the exception thrown by PostgreSQL):
'DETAIL: Key (base_id)=(1a389dc3-5bc1-4132-8a4a-c8200533503a) is not present in table "backend_base"'
"""
error_msg = ''
errors = []
# Scan all FKs
relation_fields = [f for f in obj._meta.concrete_fields if f.is_relation]
for field in relation_fields:
try:
# try to access the related target
fk_target = getattr(obj, field.name)
except ObjectDoesNotExist as e:
# Log offending FK
fk_field = field.name + '_id'
fk_value = getattr(obj, fk_field)
fk_table = field.related_model._meta.db_table
errors.append('Key (%s)=(%s) is not present in table "%s"' % (
fk_field,
str(fk_value),
fk_table
))
if len(errors):
error_msg = ' DETAIL: ' + "; ".join(errors)
return error_msg
I use airflow python operators to execute sql queries against a redshift/postgres database. In order to debug, I'd like the DAG to return the results of the sql execution, similar to what you would see if executing locally in a console:
I'm using psycop2 to create a connection/cursor and execute the sql. Having this logged would be extremely helpful to confirm the parsed parameterized sql, and confirm that data was actually inserted (I have painfully experiences issues where differences in environments caused unexpected behavior)
I do not have deep knowledge of airflow or the low level workings of the python DBAPI, but the pscyopg2 documentation does seem to refer to some methods and connection configurations that may allow this.
I find it very perplexing that this is difficult to do, as I'd imagine it would be a primary use case of running ETLs on this platform. I've heard suggestions to simply create additional tasks that query the table before and after, but this seems clunky and ineffective.
Could anyone please explain how this may be possible, and if not, explain why? Alternate methods of achieving similar results welcome. Thanks!
So far I have tried the connection.status_message() method, but it only seems to return the first line of the sql and not the results. I have also attempted to create a logging cursor, which produces the sql, but not the console results
import logging
import psycopg2 as pg
from psycopg2.extras import LoggingConnection
conn = pg.connect(
connection_factory=LoggingConnection,
...
)
conn.autocommit = True
logging.basicConfig(level=logging.DEBUG)
logger = logging.getLogger(__name__)
logger.addHandler(logging.StreamHandler(sys.stdout))
conn.initialize(logger)
cur = conn.cursor()
sql = """
INSERT INTO mytable (
SELECT *
FROM other_table
);
"""
cur.execute(sql)
I'd like the logger to return something like:
sql> INSERT INTO mytable (
SELECT ...
[2019-07-25 23:00:54] 912 rows affected in 4 s 442 ms
Let's assume you are writing an operator that uses postgres hook to do something in sql.
Anything printed inside an operator is logged.
So, if you want to log the statement, just print the statement in your operator.
print(sql)
If you want to log the result, fetch the result and print the result.
E.g.
result = cur.fetchall()
for row in result:
print(row)
Alternatively you can use self.log.info in place of print, where self refers to the operator instance.
Ok, so after some trial and error I've found a method that works for my setup and objective. To recap, my goal is to run ETL's via python scripts, orchestrated in Airflow. Referring to the documentation for statusmessage:
Read-only attribute containing the message returned by the last command:
The key is to manage logging in context with transactions executed on the server. In order for me to do this, I had to specifically set con.autocommit = False, and wrap SQL blocks with BEGIN TRANSACTION; and END TRANSACTION;. If you insert cur.statusmessage directly following a statement that deletes or inserts, you will get a response such as 'INSERT 0 92380'.
This still isn't as verbose as I would prefer, but it is a much better than nothing, and is very useful for troubleshooting ETL issues within Airflow logs.
Side notes:
- When autocommit is set to False, you must explicitly commit transactions.
- It may not be necessary to state transaction begin/end in your SQL. It may depend on your DB version.
con = psy.connect(...)
con.autocommit = False
cur = con.cursor()
try:
cur.execute([some_sql])
logging.info(f"Cursor statusmessage: {cur.statusmessage})
except:
con.rollback()
finally:
con.close()
There is some buried functionality within psycopg2 that I'm sure can be utilized, but the documentation is pretty thin and there are no clear examples. If anyone has suggestions on how to utilize things such as logobjects, or returning join PID to somehow retrieve additional information.
I need to dump the complete schema (ddl only, no data) of an Oracle database to a text file or a set of text files in order to be able to systematically track revisions to the database schema using standard VCS tools like git.
Using my favorite RDBMS, postgresql, this is an almost trivially easy task, using pg_dump --schema-only.
However, dumping an Oracle DB schema to an SQL file has proved to be a maddeningly difficult task with Oracle 11g. I'm interested to know about approaches that others have figured out.
Data pump export (no ☹)
Unfortunately, I cannot use the data pump export tools introduced in Oracle 10g, because these require DBA-level access and I cannot easily obtain this level of access for most of my clients' databases.
SQL developer
I've used Oracle's SQL developer GUI and it mostly does what I want with the "Separate files" setting:
Emits a syntactically correct SQL file to create each database object
Emits a summary SQLs file which includes each of the individual-object files in the correct order
However there are several major issues with it:
It's a GUI only; no way to script this behavior from the command line as far as I can tell
Running as an unprivileged user, it can only emit the DDL for that user's owned objects (even when that user has been granted privileges to view other users' objects ... ##$(*&!!)
It's extremely slow, taking around 20 minutes to output about 1 MB of DDL
exp and imp
Oracle's older exp command-line tool does not require DBA access. It can export the complete DDL for a database (with DBA access), or just the DDL for an individual user's owned objects.
Unfortunately, it is even slower than SQL developer (takes >1 hour for the same database even with a few performance tweaks.
However, the worst thing about exp is that it does not emit SQL, but rather a proprietary binary-format dump file (e.g. expdat.dmp).
The corresponding imp tool can "translate" these dump files into severely mangled SQL which does not contain syntactically correct end-of-statement delimiters.
Example of the horrible mangled SQL that imp show=y emits; notice the crazy line wrapping and lack of semicolons at the end of some but not all statements.
Export file created by EXPORT:V11.02.00 via direct path
import done in US7ASCII character set and AL16UTF16 NCHAR character set
import server uses AL32UTF8 character set (possible charset conversion)
. importing FPSADMIN's objects into FPSADMIN
"BEGIN "
"sys.dbms_logrep_imp.instantiate_schema(schema_name=>SYS_CONTEXT('USERENV','"
"CURRENT_SCHEMA'), export_db_name=>'*******', inst_scn=>'371301226');"
"COMMIT; END;"
"CREATE TYPE "CLOBSTRINGAGGTYPE" TIMESTAMP '2015-06-01:13:37:41' OID '367CDD"
"7E59D14CF496B27D1B19ABF051' "
"AS OBJECT"
"("
" theString CLOB,"
" STATIC FUNCTION"
" ODCIAggregateInitialize(sctx IN OUT CLOBSTRINGAGGTYPE )"
" RETURN NUMBER,"
" MEMBER FUNCTION"
" ODCIAggregateIterate(self IN OUT CLOBSTRINGAGGTYPE, VALUE IN VARC"
"HAR2 )"
" RETURN NUMBER,"
" MEMBER FUNCTION"
" ODCIAggregateTerminate(self IN CLOBSTRINGAGGTYPE, returnValue OUT"
" CLOB, flags IN NUMBER)"
" RETURN NUMBER,"
" MEMBER FUNCTION"
" ODCIAggregateMerge(self IN OUT CLOBSTRINGAGGTYPE, ctx2 IN CLOBSTR"
"INGAGGTYPE)"
" RETURN NUMBER"
");"
"GRANT EXECUTE ON "CLOBSTRINGAGGTYPE" TO PUBLIC"
"GRANT DEBUG ON "CLOBSTRINGAGGTYPE" TO PUBLIC"
"CREATE OR REPLACE TYPE BODY CLOBSTRINGAGGTYPE"
I have written a Python script to demangle the output of imp show=y, but it cannot reliably demangle the output because it doesn't understand the complete Oracle SQL syntax.
dbms_metadata
Oracle has a dbms_metadata package which supports introspection of the database contents.
It's relatively easy to write an SQL statement which will retrieve the DDL for some but not all database objects. For example, the following statement will retrieve CREATE TABLE statements, but won't retrieve the corresponding privilege GRANTs on those tables.
select sub.*, dbms_metadata.get_ddl(sub.object_type, sub.object_name, sub.owner) sql
from (
select
created,
owner,
object_name,
decode(object_type,
'PACKAGE', 'PACKAGE_SPEC',
'PACKAGE BODY', 'PACKAGE_BODY',
'TYPE BODY', 'TYPE_BODY',
object_type
) object_type
from all_objects
where owner = :un
--These objects are included with other object types.
and object_type not in ('INDEX PARTITION','LOB','LOB PARTITION','TABLE PARTITION','DATABASE LINK')
--Ignore system-generated types that support collection processing.
and not (object_type like 'TYPE' and object_name like 'SYS_PLSQL_%')
) sub
Attempting to fetch the complete set of objects quickly leads down a very complex rabbit hole. (See "Reverse engineering object DDL and finding object dependencies" for more gory details.)
What else?
Any advice? I'm at a total loss for a sane and maintainable way to perform this seemingly indispensable database programming task.
Combine DBMS_DATAPUMP, Oracle Copy (OCP), and a simple shell script to create a one-click solution.
Sample Schema to Export
--Create test user.
drop user test_user cascade;
create user test_user identified by test_user;
create table test_user.table1(a number);
create view test_user.view1 as select 1 a from dual;
create or replace procedure test_user.procedure1 is begin null; end;
/
Create Directory and Procedure
Run these as steps as SYS. The definer's rights procedure runs as SYS. This way no roles or privileges need to be granted to any users.
--Create directory that will contain SQL file.
create directory ddl_directory as 'C:\temp';
grant read on directory ddl_directory to jheller;
--Create procedure that can only export one hard-coded schema.
--This is based on René Nyffenegger's solution here:
--dba.stackexchange.com/questions/91149/how-to-generate-an-sql-file-with-dbms-datapump
create or replace procedure sys.generate_ddl authid definer is
procedure create_export_file is
datapump_job number;
job_state varchar2(20);
begin
datapump_job := dbms_datapump.open(
operation => 'EXPORT',
job_mode => 'SCHEMA',
remote_link => null,
job_name => 'Export dump file',
version => 'LATEST');
dbms_output.put_line('datapump_job: ' || datapump_job);
dbms_datapump.add_file(
handle => datapump_job,
filename => 'export.dmp',
directory => 'DDL_DIRECTORY',
filetype => dbms_datapump.ku$_file_type_dump_file);
dbms_datapump.metadata_filter(
handle => datapump_job,
name => 'SCHEMA_LIST',
value => '''TEST_USER''');
dbms_datapump.start_job(
handle => datapump_job,
skip_current => 0,
abort_step => 0);
dbms_datapump.wait_for_job(datapump_job, job_state);
dbms_output.put_line('Job state: ' || job_state);
dbms_datapump.detach(datapump_job);
end create_export_file;
procedure create_sql_file is
datapump_job number;
job_state varchar2(20);
begin
datapump_job := dbms_datapump.open(
operation => 'SQL_FILE',
job_mode => 'SCHEMA',
remote_link => null,
job_name => 'Export SQL file',
version => 'LATEST');
dbms_output.put_line('datapump_job: ' || datapump_job);
dbms_datapump.add_file(
handle => datapump_job,
filename => 'export.dmp',
directory => 'DDL_DIRECTORY',
filetype => dbms_datapump.ku$_file_type_dump_file);
dbms_datapump.add_file(
handle => datapump_job,
filename => 'schema.sql',
directory => 'DDL_DIRECTORY',
filetype => dbms_datapump.ku$_file_type_sql_file);
dbms_datapump.start_job(
handle => datapump_job,
skip_current => 0,
abort_step => 0);
dbms_datapump.wait_for_job(datapump_job, job_state);
dbms_output.put_line('Job state: ' || job_state);
dbms_datapump.detach(datapump_job);
end create_sql_file;
begin
create_export_file;
create_sql_file;
end;
/
--Grant to users.
grant execute on generate_ddl to jheller;
Setup OCP on the Client
Files on an Oracle directory can be easily transferred to a client PC using OCP as described in this answer. The setup
is a bit tricky - download the precise version of the program and the instant client and unzip them into the same directory. I think I also had some problems with
a VC++ redistributable or something the first time.
Commands to Run
Now the easy part - creating and moving the files is done in two simple steps:
execute sys.generate_ddl;
C:\Users\jonearles\Downloads\ocp-0.1-win32>ocp jheller/jheller#orcl12 DDL_DIRECTORY:schema.sql schema.sql
Sample Output
This script contains a lot of weird things. Some weird extra commands that nobody will understand, and some weird options that nobody will understand. That's probably one of the reasons this seemingly obvious feature is so difficult - due to the thousands of odd features, it's impossible to have output that is both understandable and completely unambiguous.
CREATE TABLE "TEST_USER"."TABLE1"
( "A" NUMBER
) SEGMENT CREATION DEFERRED
PCTFREE 10 PCTUSED 40 INITRANS 1 MAXTRANS 255
NOCOMPRESS LOGGING
TABLESPACE "USERS" ;
...
-- new object type path: SCHEMA_EXPORT/PROCEDURE/PROCEDURE
-- CONNECT TEST_USER
CREATE EDITIONABLE procedure procedure1 is begin null; end;
/
...
-- new object type path: SCHEMA_EXPORT/VIEW/VIEW
CREATE FORCE EDITIONABLE VIEW "TEST_USER"."VIEW1" ("A") AS
select 1 a from dual
;
How do you have created a status of blog, micropost and etc?
I simply have created a status which each application entity has a status code of boolean.
False is private, true is public.
It is a multi-user environment in fosuserbundle.
It was created without any problems until now.
However, I wanted to be able to set user, group, and other status set in a multi-user environment.
(like a linux permissions, 744, etc...)
So, I think that it might create a bundle that manages the status.
The comment bundle that I have created is relation each application bundle by the ID of the string.
{{ render(controller('MyCommentBundle:Default:embed', {
'id' : 'for_example_micropost_' ~ micropost.id, // => relation id
'name' : micropost.owner.username,
'request': app.request
})) }}
However, it can not use when viewing list of application contents.
(Because the query occurs in one by one)
For example, either simply relationship by creating a column for each application.
Status Entity
blog_id
micropost_id
picture_id
etc...
But I think it inconvenient.
It require care of additional column when generated for each new application.
And I feel it is not a smart also.
For example, do so in the list part. Conditions of the string if possible in DQL.
public function find_all($owner, $limit, $offset)
{
$q = $this->getEntityManager()->createQuery('
SELECT p FROM MyMicropostBundle:Post p
LEFT JOIN MyStatusBundle s
With s.relation_id = :prefix + p.id
WHERE s.code > 744
')
->setParameter("prefix", 'for_example_micropost_')
->setParameter("owner", $owner)
->setMaxResults($limit)
->setFirstResult($offset);
return $q->getResult();
}
Etc., various I think, but I do not know the best way.
How do you have created a status?
Do you know status bundle already?
Or do you know bundle may be relevant to me.
If there is an optimal way and ideas, please tell me.
StatusBundle process:
It check a access user first.
It check a status code and groups and owner associated with the contents.
It compare a status code, groups and owner of the content to a access user.
I think it is possible to create (relation) bundle within a single page in my skills.
However, problem is how to relations at such a list display page that query to get the contents of the large number.
For example, I get the data of the last 10 blog posts.
I get the last 10 of public status posts if a access user is not the owner.
It is easy if Post entity has a status.
However, I think how to check a status, groups and owner if a status of a post has been registered in the status entity of other bundle.
I think that how to handle the status bundle while I get the data of the contents of the large number.