Can anyone guide me if am doing anything wrong :
Objective : Want to set up scheduler HA .
Versions : Backend db - Postgres 12.6 , Airflow 2.1.1
Challenges: When scheduler is started on first machine , it works as expected and i was able to trigger the example _bash_operator but when scheduler is started on another host with the same backend connection .
My first scheduler fails and it gives me the below error when am trying to click on the bash_oprator_example dag in WebUI
ValueError: unsupported pickle protocol: 5
ValueError: unsupported pickle protocol: 5 generally occurs when you have a different version of Python running on both machines.
Verify that you have same versions of Python on both machines
Related
We have a MySQL task that is connecting to a 5.7.22 MySQL DB just fine in Airflow 1.10.15 using SSL (Extra: {"ssl": {"ca": "ca.pem"}} ). However, once we upgraded to Airflow 2.0.2 the task fails with (2026, 'SSL connection error: error:1425F102:SSL routines:ssl_choose_client_version:unsupported protocol'). Does anyone have any ideas about this behavior?
Below are the python packages for both before and after
Airflow 1.10.15
mysqlclient==1.3.14
Airflow 2.0.2
mysql-connector-python==8.0.22
mysqlclient==2.0.3
I think the problem might be because Airflow 2 uses a newer Debian which does not support TLS v1.
I am not 100% sure how you should configure your MySQL/Client but this sounds awfully similar to MySQL ERROR 2026 - SSL connection error - Ubuntu 20.04
The solution is either to disable SSL (not recommended) or decrease security level of Debian (not recommended) or make sure that you use TLS v2
Apparently this can be done via changing your MySQL seetings and restarting it:
[mysqld]
tls_version=TLSv1.2
I'm trying to run airflow with Azure SQL database as backend using mssql+pyodbc connection string(all relevant drivers have been installed).
while airflow is able to connect to DB and create tables, i.e, airflow initdb runs successfully, I'm facing issues while running airflow scheduler, as a result, the tasks triggered are always in "running" state.
This is the error I get while running airflow scheduler:
*sqlalchemy.exc.ProgrammingError: (pyodbc.ProgrammingError) ('42000', "[42000] [Microsoft][ODBC Driver 17 for SQL Server][SQL Server]Incorrect syntax near '1'. (102) (SQLExecDirectW)")
[SQL: SELECT dag.dag_id AS dag_dag_id
FROM dag
WHERE dag.is_paused IS 1 AND dag.dag_id IN (?)]
[parameters: ('example_http_operator',)]*
(Background on this error at: http://sqlalche.me/e/13/f405)
I'm using apache-airflow==1.10.11.
If you were able to run airflow + azure SQL DB with any configuration please feel free to jump in.
I found a document and talk the configuration about run airflow + azure SQL DB. Maybe it's helpful for you.
Ref: Setting up Airflow on Azure & connecting to MS SQL Server
This post also give some configurations about it: Apache Airflow - Connection issue to MS SQL Server using pymssql + SQLAlchemy
For MSSQL as backend DB, there is workaround in Airflow#10713. I using apache-airflow==1.10.15 and solved same error as yours.
The command suggested is attached, but I use vi update instead of run sed command.
RUN sed -i 's/import copy/import copy,sqlalchemy/g' /usr/local/lib/python3.6/site-packages/airflow/models/dag.py \ && sed -i 's/DagModel.is_paused.is_(True)/DagModel.is_paused == sqlalchemy.sql.expression.true()/g' /usr/local/lib/python3.6/site-packages/airflow/models/dag.py
Airflow version: 1.10.9
Executor : LocalExecutor
Docket Setup
when job runs sometime we are getting following error. I have searched in web, many people faced this issue in celeryExecutor but we are using LocalExecutor(Docker setup). How can I resolve this problem?
*** Log file does not exist: /home/ubuntu/airflow/airflow/logs/es_update_relevance_score/es_update_relevance_score/2020-05-14T16:26:06.062416+00:00/1.log
*** Fetching from: http://:8793/log/es_update_relevance_score/es_update_relevance_score/2020-05-14T16:26:06.062416+00:00/1.log
*** Failed to fetch log file from worker. Invalid URL 'http://:8793/log/es_update_relevance_score/es_update_relevance_score/2020-05-14T16:26:06.062416+00:00/1.log': No host supplied
Here is one approach I've seen when running the scheduler and webserver in their own containers and using LocalExecutor:
Mount a host log directory as a volume into both the scheduler and webserver containers:
volumes:
- /location/on/host/airflow/logs:/opt/airflow/logs
Make sure the user within the airflow containers (usually airflow) has permissions to read and write that directory. If the permissions are wrong you will see an error like the one in your post.
This probably won't scale beyond LocalExecutor usage though.
I'm trying to set up Apache Kafka to communicate between two virtual machines running CentOS on the same network. I originally set up a Kafka producer and consumer on one machine and everything was running smoothly. I then set up Kafka on the other machine and in the process of trying to get them to connect, I get the error "bootstrap-server is not a recognized option" (I'm running the most recent version of Kafka, 2.2).
This is what I used to attempt a producer connection:
bin/kafka-console-producer.sh --bootstrap-server 10.0.0.11:9092 --topic test
And on the consumer side:
bin/kafka-console-producer.sh --bootstrap-server 10.0.0.11:9092 --topic test
The 10.0.0.11 machine is running the server itself.
According do Apache Kafka Documentation, that can be found here: https://kafka.apache.org/documentation/#quickstart_send, you should use --broker-list property to pass broker address.
Command will be:
./bin/kafka-console-producer.sh --broker-list localhost:9092 --topic test
EDIT
From Apache Kafka 2.5 both options are supported --broker-list and --bootstrap-server. Suggested one is --bootstrap-server
Following command should work.
./bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic test
./kafka-console-consumer.sh --zookeeper localhost:2181 --topic fred
Simply giving this command worked. However in upcoming major release using the option bootstrap-server will be mandatory.
When I use the above command it gives a warning: "Using the ConsoleConsumer with old consumer is deprecated and will be removed in a future major release. Consider using the new consumer by passing [bootstrap-server] instead of [zookeeper]."
I am running Airflowv1.9 with Celery Executor. I have 5 Airflow workers running in 5 different machines. Airflow scheduler is also running in one of these machines. I have copied the same airflow.cfg file across these 5 machines.
I have daily workflows setup in different queues like DEV, QA etc. (each worker runs with an individual queue name) which are running fine.
While scheduling a DAG in one of the worker (no other DAG have been setup for this worker/machine previously), I am seeing the error in the 1st task and as a result downstream tasks are failing:
*** Log file isn't local.
*** Fetching here: http://<worker hostname>:8793/log/PDI_Incr_20190407_v2/checkBCWatermarkDt/2019-04-07T17:00:00/1.log
*** Failed to fetch log file from worker. 404 Client Error: NOT FOUND for url: http://<worker hostname>:8793/log/PDI_Incr_20190407_v2/checkBCWatermarkDt/2019-04-07T17:00:00/1.log
I have configured MySQL for storing the DAG metadata. When I checked task_instance table, I see proper hostnames are populated against the task.
I also checked the log location and found that the log is getting created.
airflow.cfg snippet:
base_log_folder = /var/log/airflow
base_url = http://<webserver ip>:8082
worker_log_server_port = 8793
api_client = airflow.api.client.local_client
endpoint_url = http://localhost:8080
What am I missing here? What configurations do I need to check additionally for resolving this issue?
Looks like the worker's hostname is not being correctly resolved.
Add a file hostname_resolver.py:
import os
import socket
import requests
def resolve():
"""
Resolves Airflow external hostname for accessing logs on a worker
"""
if 'AWS_REGION' in os.environ:
# Return EC2 instance hostname:
return requests.get(
'http://169.254.169.254/latest/meta-data/local-ipv4').text
# Use DNS request for finding out what's our external IP:
s = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
s.connect(('1.1.1.1', 53))
external_ip = s.getsockname()[0]
s.close()
return external_ip
And export: AIRFLOW__CORE__HOSTNAME_CALLABLE=airflow.hostname_resolver:resolve
The web program of the master needs to go to the worker to fetch the log and display it on the front-end page. This process is to find the host name of the worker. Obviously, the host name cannot be found,Therefore, add the host name to IP mapping on the master's vim /etc/hosts
If this happens as part of a Docker Compose Airflow setup, the hostname resolution needs to be passed to the container hosting the webserver, e.g. through extra_hosts:
# docker-compose.yml
version: "3.9"
services:
webserver:
extra_hosts:
- "worker_hostname_0:192.168.xxx.yyy"
- "worker_hostname_1:192.168.xxx.zzz"
...
...
More details here.