I have the following task in my DAG:
create_features_table = PostgresOperator(
task_id="create_features_table",
postgres_conn_id="featuredb",
sql="src/test.sql "
)
But when I test the task, I get this error:
psycopg2.errors.SyntaxError: syntax error at or near "src"
LINE 1: src/test.sql
The content of the test.sql script is:
CREATE TABLE test(
C1 int,
C2 int,
);
I can't point out the error in the syntax, but that's because it is my first DAG. Any help would be greatly appreciated.
If I run the script directly from the postgres container's psql using "\i src/text.sql" it works fine.
I have tested the connection from the airflow web server and the connection works.
I found that I had to put a space before closing the quotes to avoid a jinja2.exeptions.TemplateNotFound error, but haven't been able to find the syntax error.
According to documentation (https://airflow.apache.org/docs/apache-airflow-providers-postgres/stable/_api/airflow/providers/postgres/operators/postgres/index.html#airflow.providers.postgres.operators.postgres.PostgresOperator)
if you are defining your sql script path, it must ends with .sql
You have a white space in your path in the end so Operatort things it’s a query to be executed on your postgre, not file with query. You can see it in the response from postgre. Run this query on your postgre instance src/test.sql and you will get the same syntax error.
You can fix it easily by removing that white space
sql="src/test.sql"
Related
If I run this locally in the cli it runs successfully and copies the files from another bucket/key to mine into the correct location.
aws s3 sync s3://client_export/ref/commissions/snapshot_date=2022-01-01/ s3://bi-dev/KSM/refinery29/commissions/snapshot_date=2022-01-01/
When I try with the S3CopyObjectOperator I see the NoSuchKey error:
copy_commissions_data = S3CopyObjectOperator(
task_id='copy_commissions_data',
aws_conn_id='aws_default',
source_bucket_name='client_export',
dest_bucket_name='bi-dev',
source_bucket_key='ref/commissions/snapshot_date=2022-01-01,
dest_bucket_key='KSM/refix/commissions/snapshot_date=2022-01-01',
dag=dag
)
I've also tried adding a / before and after the key names and both but I get the same error
You are missing quote in the end of line 6, should be:
source_bucket_key='ref/commissions/snapshot_date=2022-01-01',
I'm trying to connect to the multiple databases and create tables, but when migrating flyway gets syntax error.
This is the migration file I'm trying to run:
\c testdatabase;
CREATE TABLE testtable1;
\c testdatabase2;
CREATE TABLE testtable2;
Flyway gives this output:
Error Code : 0
Message : ERROR: syntax error at or near "\"
Position: 1
Line : 1
Statement : \c testdatabase
It seems like flyway does not support meta-commands like "\c" for connecting to the database. Is there any other way to do connect to the databases and create a table?
The error comes (as indicated in the error input) from the comment lines preceding your two SQL statements in the script: \c testdatabase; which are not valid SQL syntax for comments.
You could simply correct those faulty lines like the following: -- testdatabase, and generally, the error input already gives you a hint as to where lies the problem.
I want to execute a batch file using People code in Application Engine Program. But The program have an issue returning Exec code as a non zero value (Value - 1).
Below is people code snippet below.
Global File &FileLog;
Global string &LogFileName, &Servername, &commandline;
Local string &Footer;
If &Servername = "PSNT" Then
&ScriptName = "D: && D:\psoft\PT854\appserv\prcs\RNBatchFile.bat";
End-If;
&commandline = &ScriptName;
/* Need to commit work or Exec will fail */
CommitWork();
&ExitCode = Exec("cmd.exe /c " | &commandline, %Exec_Synchronous + %FilePath_Absolute);
If &ExitCode <> 0 Then
MessageBox(0, "", 0, 0, ("Batch File Call Failed! Exit code returned by script was " | &ExitCode));
End-If;
Any help how to resolve this issue.
Best bet is to do a trace of the execution.
Thoughts:
Can you log on the the process scheduler you are running this on and execute the script OK?
Is the AE being scheduled or called at run-time?
You should not need to change directory as you are using a fully qualified path to the script.
you should not need to call "cmd /c" as this will create an additional shell for you application to run within, making debuging harder, etc.
Run a trace, and drop us the output. :) HTH
What about changing the working directory to D: inside of the script instead? You are invoking two commands and I'm wondering what the shell is returning to exec. I'm assuming you wrote your script to give the appropriate return code and that isn't the problem.
I couldn't tell from the question text, but are you looking for a negative result, such as -1? I think return codes are usually positive. 0 for success, some other positive number for failure. Negative numbers may be acceptable, but am wondering if Exec doesn't like negative numbers?
Perhaps the PeopleCode ChDir function still works as an alternative to two commands in one line? I haven't tried it for a LONG time.
Another alternative that gives you significant control over the process is to use java.lang.Runtime.exec from PeopleCode: http://jjmpsj.blogspot.com/2010/02/exec-processes-while-controlling-stdin.html.
Background: I am developing a rscript that pulls data from a mysql database, performs a logistic regression and then inserts the predictions back into the database. I want the entire system to be self contained in the script in case of database failure. This includes all mysql stored procedures that the script depends on to aggregate the data on the backend since these would be deleted in such a database failure.
Question: I'm having trouble creating a stored procedure from an R script. I am running the following:
mySQLDriver <- dbDriver("MySQL")
connect <- dbConnect(mySQLDriver, group = connection)
query <-
"
DROP PROCEDURE IF EXISTS Test.Tester;
DELIMITER //
CREATE PROCEDURE Test.Tester()
BEGIN
/***DO DATA AGGREGATION***/
END //
DELIMITER ;
"
sendQuery <- dbSendQuery(connect, query)
dbClearResult(dbListResults(connect)[[1]])
dbDisconnect(connect)
I however get the following error that seems to involve the DELIMITER change.
Error in .local(conn, statement, ...) :
could not run statement: You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near 'DELIMITER //
CREATE PROCEDURE Test.Tester()
BEGIN
/***DO DATA AGGREGATION***/
EN' at line 2
What I've Done: I have spent quite a bit of time searching for the answer, but have come up with nothing. What am I missing?
Just wanted to follow up on this string of comments. Thank you for your thoughts on this issue. I have a couple Python scripts that need to have this functionality and I began researching the same topic for Python. I found this question that indicates the answer. The question states:
"The DELIMITER command is a MySQL shell client builtin, and it's recognized only by that program (and MySQL Query Browser). It's not necessary to use DELIMITER if you execute SQL statements directly through an API.
The purpose of DELIMITER is to help you avoid ambiguity about the termination of the CREATE FUNCTION statement, when the statement itself can contain semicolon characters. This is important in the shell client, where by default a semicolon terminates an SQL statement. You need to set the statement terminator to some other character in order to submit the body of a function (or trigger or procedure)."
Hence the following code will run in R:
mySQLDriver <- dbDriver("MySQL")
connect <- dbConnect(mySQLDriver, group = connection)
query <-
"
CREATE PROCEDURE Test.Tester()
BEGIN
/***DO DATA AGGREGATION***/
END
"
sendQuery <- dbSendQuery(connect, query)
dbClearResult(dbListResults(connect)[[1]])
dbDisconnect(connect)
I've written a simple trigger using HDBC and Sqlite3.
calculateNoOfStocksTraded::Database.HDBC.Sqlite3.Connection->IO Integer
calculateNoOfStocksTraded conn=do
run conn "CREATE TRIGGER calStocks\
\AFTER INSERT ON historicalData\
\FOR EACH ROW\
\BEGIN\
\UPDATE company\
\SET noOfStocks=300.0;\
\END " []
It keeps saying that there's a syntax error. I don't know how to figure it out. pls help me to locate the error
The pieces of your SQL strings are being run together without spaces between them (because that's how the string breaks with \ work in Haskell). Try adding a space at the end of each line of the query, before the \.