Running Apache Airflow with Azure SQL server as backend DB - odbc

I'm trying to run airflow with Azure SQL database as backend using mssql+pyodbc connection string(all relevant drivers have been installed).
while airflow is able to connect to DB and create tables, i.e, airflow initdb runs successfully, I'm facing issues while running airflow scheduler, as a result, the tasks triggered are always in "running" state.
This is the error I get while running airflow scheduler:
*sqlalchemy.exc.ProgrammingError: (pyodbc.ProgrammingError) ('42000', "[42000] [Microsoft][ODBC Driver 17 for SQL Server][SQL Server]Incorrect syntax near '1'. (102) (SQLExecDirectW)")
[SQL: SELECT dag.dag_id AS dag_dag_id
FROM dag
WHERE dag.is_paused IS 1 AND dag.dag_id IN (?)]
[parameters: ('example_http_operator',)]*
(Background on this error at: http://sqlalche.me/e/13/f405)
I'm using apache-airflow==1.10.11.
If you were able to run airflow + azure SQL DB with any configuration please feel free to jump in.

I found a document and talk the configuration about run airflow + azure SQL DB. Maybe it's helpful for you.
Ref: Setting up Airflow on Azure & connecting to MS SQL Server
This post also give some configurations about it: Apache Airflow - Connection issue to MS SQL Server using pymssql + SQLAlchemy

For MSSQL as backend DB, there is workaround in Airflow#10713. I using apache-airflow==1.10.15 and solved same error as yours.
The command suggested is attached, but I use vi update instead of run sed command.
RUN sed -i 's/import copy/import copy,sqlalchemy/g' /usr/local/lib/python3.6/site-packages/airflow/models/dag.py \ && sed -i 's/DagModel.is_paused.is_(True)/DagModel.is_paused == sqlalchemy.sql.expression.true()/g' /usr/local/lib/python3.6/site-packages/airflow/models/dag.py

Related

Airflow Scheduler HA

Can anyone guide me if am doing anything wrong :
Objective : Want to set up scheduler HA .
Versions : Backend db - Postgres 12.6 , Airflow 2.1.1
Challenges: When scheduler is started on first machine , it works as expected and i was able to trigger the example _bash_operator but when scheduler is started on another host with the same backend connection .
My first scheduler fails and it gives me the below error when am trying to click on the bash_oprator_example dag in WebUI
ValueError: unsupported pickle protocol: 5
ValueError: unsupported pickle protocol: 5 generally occurs when you have a different version of Python running on both machines.
Verify that you have same versions of Python on both machines

Azure Cosmos DB Emulator Not running - It Starts and then throws Error

I am trying to install and run Cosmos DB Emulator on my machine but it is not letting me to connect to Azure Cosmos DB Emulator. When I run the Azure Cosmos DB Emulator, it shows the "Started" notification and then opened a page in browser but the page is not loaded. I am tired of doing everything which I have found on google.
Here is the Azure Cosmos DB Emulator Error:
And when the page is opened on link https://localhost:8081/_explorer/index.html, the Firefox browser shows "Unable to connect"
The strange thing is, when I installed it on another machine, it ran there without any issue.
After full of 3 hectic days, I have successfully run Cosmos DB Emulator. Here is the work around.
Shut down Azure Cosmos DB Emulator
I tried to run lodctr /R in command line by running it as "Run as Administrator", it threw an error: "Error: Unable to rebuild performance counter setting from system backup store, error code is 2"
Found a solution for the lodctr /R error which is to run another command as below:
"c:\Windows\Microsoft.NET\Framework\v2.0.50727\aspnet_regiis.exe -i"
It will install ASP.NET (2.0.50727). Here is the link: https://www.stevefenton.co.uk/2016/04/unable-to-rebuild-performance-counter-setting-from-system-backup-store/
Then again run the command lodctr /R, it sets the counter
Then start Azure Cosmos DB Emulator and it will run.
I used to get this constantly, I realised I was shutting down my pc before stopping the emulator. Once I started stopping the emulator before shutdown, this issue went away for me.
There are 5 ports the emulator runs on out of the box: 10251,10252,10253,10254. You can confirm this by typing:
.\CosmosDB.Emulator.exe -h
And you'll see:
/DirectPorts= Comma-separated list of 4 ports to use for direct connectivity. Default is 10251,10252,10253,
10254.
What tends to be my issue is that the emulator is still running on one of the above ports but .\CosmosDB.Emulator.exe /Shutdown doesn't effectively stop the process.
To combat this, I search to see which port is active:
PS C:\Program Files\Azure Cosmos DB Emulator> netstat -ano | findstr :10251
PS C:\Program Files\Azure Cosmos DB Emulator> netstat -ano | findstr :10252
PS C:\Program Files\Azure Cosmos DB Emulator> netstat -ano | findstr :10253
TCP 127.0.0.1:10253 0.0.0.0:0 LISTENING 27772
And then I stop that process:
stop-process 27772 // this is using powershell but that's unimportant
Then I can start the emulator again without issue:
.\CosmosDB.Emulator.exe

Airflow: New DAG is not found by webserver

In Airflow, how should I handle the error "This DAG isn't available in the webserver DagBag object. It shows up in this list because the scheduler marked it as active in the metadata database"?
I've copied a new DAG to an Airflow server, and have tried:
unpausing it and refreshing it (basic operating procedure, given in this previous answer https://stackoverflow.com/a/42291683/160406)
restarting the webserver
restarting the scheduler
stopping the webserver and scheduler, resetting the database (airflow resetdb), then starting the webserver and scheduler again
running airflow backfill (suggested here Airflow "This DAG isnt available in the webserver DagBag object ")
running airflow trigger_dag
The scheduler log shows it being processed and no errors occurring, I can interact with it and view it's state through the CLI, but it still does not appear in the web UI.
Edit: the webserver and scheduler are running on the same machine with the same airflow.cfg. They're not running in Docker.
They're run by Supervisor, which runs them both as the same user (airflow). The airflow user has read, write and execute permission on all of the dag files.
This helped me...
pkill -9 -f "airflow scheduler"
pkill -9 -f "airflow webserver"
pkill -9 -f "gunicorn"
then restart the airflow scheduler and webserver.
Just had this issue myself. After changing permissions, resetting the meta database, restarting the webserver & even making some potential code changes to rectify the situation, it didn't happen.
However, I noticed that even though we were stopping the webserver, our gunicorn process was still running. Killing these processes & then starting everything back up resulted in success
I had the same problem on an airflow installed from a Docker image
What I did was:
1- delete all files .pyc
2- delete Metadata databse using :
for t in ["xcom", "task_instance", "sla_miss", "log", "job", "dag_run", "dag" ]:
sql="delete from {} where dag_id='{}'".format(t, dag_input)
hook.run(sql, True)
3- restart webserver & scheduler
4- Execute airflow updatedb
It resolved the problem for me.
if the airflow_home - dags_folder config parameter is same for scheduler, webUI and the command line interface the only cause for the error:
This DAG isn't available in the webserver DagBag object
can be file permissions or error in python script.
Please check
Run the dag as normal python script and check for errors
User in airflow.cfg and the one creating the dag should be same or the dag should have execute permission for the airflow user
With Airflow 1.9 I don't experience the problem with zombie gunicorn processes.
I do a simple restart: systemctl restart airflow-webserver and it forces webserver to refresh DAG status.

Oracle 11g XE installation on docker RHEL 7 image

While installing oracle 11g XE on docker i am getting the error.
Following are the output:-
/etc/init.d/oracle-xe configure
Oracle Database 11g Express Edition Configuration
This will configure on-boot properties of Oracle Database 11g Express
Edition. The following questions will determine whether the database should
be starting upon system boot, the ports it will use, and the passwords that
will be used for database accounts. Press to accept the defaults.
Ctrl-C will abort.
Specify the HTTP port that will be used for Oracle Application Express [8080]:8080
Specify a port that will be used for the database listener [1521]:1521
Specify a password to be used for database accounts. Note that the same
password will be used for SYS and SYSTEM. Oracle recommends the use of
different passwords for each database account. This can be done after
initial configuration:
Confirm the password:
Do you want Oracle Database 11g Express Edition to be started on boot (y/n) [y]:y
Starting Oracle Net Listener...Done
Configuring database...
Database Configuration failed. Look into /u01/app/oracle/product/11.2.0/xe/config/log for details
[root#b7c63c4e1da8 Disk1]# cd /u01/app/oracle/product/11.2.0/xe/config/log
[root#b7c63c4e1da8 log]# ls
CloneRmanRestore.log cloneDBCreation.log postDBCreation.log postScripts.log
[root#b7c63c4e1da8 log]# cat CloneRmanRestore.log
ORA-00845: MEMORY_TARGET not supported on this system
select TO_CHAR(systimestamp,'YYYYMMDD HH:MI:SS') from dual
*
ERROR at line 1:
ORA-01034: ORACLE not available
Process ID: 0
Session ID: 0 Serial number: 0
declare
*
ERROR at line 1:
ORA-01034: ORACLE not available
Process ID: 0
Session ID: 0 Serial number: 0
select TO_CHAR(systimestamp,'YYYYMMDD HH:MI:SS') from dual
*
ERROR at line 1:
ORA-01034: ORACLE not available
Process ID: 0
Session ID: 0 Serial number: 0
One of the possible solution that I got was to mount the temp file to provide extra space to it which only contains 6GB approx in the docker. But i am unable to mount the memory in docker.
Got the solution for the same :-
we have to modify the files init.ora and initXETemp.ora at the path /u01/app/oracle/product/11.2.0/xe/config/scripts
with the values :-
###########################################
# Miscellaneous
###########################################
compatible=11.2.0.0.0
diagnostic_dest=/u01/app/oracle
#memory_target=1073741824
pga_aggregate_target=200540160
sga_target=601620480
You may encounter
ORA-00845: MEMORY_TARGET not supported on this system
when starting Oracle DB in an unprivileged container. Try running the container with the --privileged flag, e.g.
docker run --name oracle12 --hostname oracledb --privileged local/oracle12:12.1.0.2

sqoop query to get sql server data into cloudera manager

sqoop import --connect 'jdbc:sqlserver://IP address;username=user;password=pswd;database=Master' --table [Person].[BusinessEntityContact] --target-dir /home/ubuntu/hdfs/dir is not working .
Reference:http://mapredit.blogspot.com/2011/10/sqoop-and-microsoft-sql-server.html [1]: http://i.stack.imgur.com/W5mBB.png
In your error logs got SQLServerException and says "Connect timed out. Verify the connection properties.". Please check whether you have access from where you try run this command and also MSSQL port "1433". Then add number of maps "-m" in your command.
Nirmale
can you try from your unix box to
curl http://131.107.174.121:1433
If you get "Empty reply from server" it is ok, or if you get an error like "couldn't connect to host", check with your SQL Server admin as to what port the SQL Server is listening on.
Best way to check is using sqoop list-tables command, something as follows:
sqoop list-tables -connect 'jdbc:sqlserver://IP address;username=user;password=pswd;database=Master' -username --password

Resources