I installed airflow within one of my Anaconda envs named engdados. When I execute the command airflow initdb I'm getting the following error: airflow initdb: cannot import name 'Pendulum' from 'pendulum'. The full trace back is shown below:
(engdados) guilherme#Athena-LNX:~$ airflow initdb
Traceback (most recent call last):
File "/home/guilherme/anaconda3/envs/engdados/bin/airflow", line 25, in <module>
from airflow.configuration import conf
File "/home/guilherme/anaconda3/envs/engdados/lib/python3.8/site-packages/airflow/__init__.py", line 47, in <module>
settings.initialize()
File "/home/guilherme/anaconda3/envs/engdados/lib/python3.8/site-packages/airflow/settings.py", line 403, in initialize
configure_adapters()
File "/home/guilherme/anaconda3/envs/engdados/lib/python3.8/site-packages/airflow/settings.py", line 319, in configure_adapters
from pendulum import Pendulum
ImportError: cannot import name 'Pendulum' from 'pendulum' (/home/guilherme/anaconda3/envs/engdados/lib/python3.8/site-packages/pendulum/__init__.py)
(engdados) guilherme#Athena-LNX:~$ service start mysql$
start: unrecognized service
(engdados) guilherme#Athena-LNX:~$ service mysql start$
Usage: /etc/init.d/mysql start|stop|restart|reload|force-reload|status|bootstrap
(engdados) guilherme#Athena-LNX:~$ airflow initdb
Traceback (most recent call last):
File "/home/guilherme/anaconda3/envs/engdados/bin/airflow", line 25, in <module>
from airflow.configuration import conf
File "/home/guilherme/anaconda3/envs/engdados/lib/python3.8/site-packages/airflow/__init__.py", line 47, in <module>
settings.initialize()
File "/home/guilherme/anaconda3/envs/engdados/lib/python3.8/site-packages/airflow/settings.py", line 403, in initialize
configure_adapters()
File "/home/guilherme/anaconda3/envs/engdados/lib/python3.8/site-packages/airflow/settings.py", line 319, in configure_adapters
from pendulum import Pendulum
ImportError: cannot import name 'Pendulum' from 'pendulum' (/home/guilherme/anaconda3/envs/engdados/lib/python3.8/site-packages/pendulum/__init__.py)
The problem is: the pendulum is installed! When I execute the conda list command I can see the Pendulum there as follows:
Name Version Build Channel
pendulum 2.1.2 pypi_0 pypi
What I've checked so far:
Is the engdados environment activated? Yes
Is the Pendulum installed on Anaconda environment? Yes
The version of Pendulum the Anaconda shows is different of the one showed in conda list (1.4.4). Why?
I have no idea what is going on. Thanks in advance.
In pendulum version 2, the class pendulum.Pendulum is replaced with pendulum.DateTime.
Your version of airflow is expecting pendulum 1.x but your environment has 2.x.
You may be able to fix this by making a new env and installing airflow 2.0 (which uses pendulum 2.x). If you must use airflow < 2.0, you will need to pin pendulum to < 2.0 (e.g. using pip constraints).
Also if you using Pendulum in your code, for example in custom Operators you can add
try:
from pendulum import DateTime as Pendulum
except ImportError:
from pendulum import Pendulum
Related
when I start my airflow schedular and webserver my bigdata.py file not getting imported, below is the error which am getting.
​Broken DAG: [/home/adminn/airflow/dags/bigdata.py] Traceback (most recent call last):
File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
File "<frozen importlib._bootstrap>", line 1004, in _find_and_load_unlocked
ModuleNotFoundError: No module named 'airflow.providers.apache'
this is the DAG I have written, am I missing something?
here I am trying to pull MySQL table using sqoop and load in HDFS, and scheduling this operation using Airflow.
from airflow.models import DAG
from airflow.contrib.operators.sqoop_operator import SqoopOperator
from airflow.utils.dates import days_ago
Dag_Sqoop_Import = DAG(dag_id="SqoopImport",
schedule_interval="* * * * *",
start_date=days_ago(2))
sqoop_mysql_import = SqoopOperator(conn_id="sqoop_local",
table="shipmethod",
cmd_type="import",
target_dir="/airflow_sqoopImport",
num_mappers=1,
task_id="SQOOP_Import",
dag=Dag_Sqoop_Import)
sqoop_mysql_import
replies appreciated,thanks.
From airflow 2.x the providers are no longer included by default, but you have to install and then import them and the import path is changed
In your case you have to:
install the sqoop provider by running: pip install 'apache-airflow-providers-apache-sqoop'
change the import string for the operator into: from airflow.providers.apache.sqoop.operators.sqoop import SqoopOperator
Here is the full list of the providers available:
https://airflow.apache.org/docs/apache-airflow-providers/packages-ref.html
And this is the provider you are looking for:
https://airflow.apache.org/docs/apache-airflow-providers/packages-ref.html#apache-airflow-providers-apache-sqoop
Trying to import the mssql hook and operator into my dag but I keep getting this error from Airflow.
I'm currently importing with the newest syntax:
from airflow.providers.microsoft.mssql.hooks.mssql import MsSqlHook
from airflow.providers.microsoft.mssql.operators.mssql import MsSqlOperator
and I'm getting this import error:
File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
File "/opt/airflow/dags/hevo_dag.py", line 5, in <module>
from airflow.providers.microsoft.mssql.hooks.mssql import MsSqlHook
ModuleNotFoundError: No module named 'airflow.providers.microsoft.mssql'
To import the operator you need to do:
For Airflow>=2.0:
you need to use Mssql provider package:
pip install apache-airflow-providers-microsoft-mssql
For Airflow<2.0:
you need to use Mssql backport provider package:
pip install apache-airflow-backport-providers-microsoft-mssql
In your code it's the same import path (regardless of the package). Airflow backported the provider to ease migration from Airflow 1 to Airflow 2 so upon upgrading you will not need to change the import paths.
I have written a Dag file with contain ssh hook implementation.
I get import error in the DAG file
File "/usr/local/lib/python3.6/dist-packages/paramiko/transport.py", line 91, in
from paramiko.dsskey import DSSKey
File "/usr/local/lib/python3.6/dist-packages/paramiko/dsskey.py", line 25, in
from cryptography.hazmat.primitives import hashes, serialization
ImportError: cannot import name 'serialization'
how to resolve this?
thanks in advance
Sundar
I have problem with starting my jupyter notebook after uninstalling and updating of sklearn library
as below.
conda uninstall scikit-learn --yes
conda install scikit-learn==0.23.1
I can not start my jupyter notebook from ANACONDA NAGIGATOR by clicking the jupyter notebook launch button.
The Application launch error says,
Traceback (most recent call last):
File "D:\Anaconda3\lib\site-packages\jsonschema\__init__.py", line 31, in
from importlib import metadata
ImportError: cannot import name 'metadata' from 'importlib' (D:\Anaconda3\lib\importlib\__init__.py)
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "D:\Anaconda3\Scripts\jupyter-notebook-script.py", line 6, in
from notebook.notebookapp import main
File "D:\Anaconda3\lib\site-packages\notebook\notebookapp.py", line 85, in
from .services.contents.manager import ContentsManager
File "D:\Anaconda3\lib\site-packages\notebook\services\contents\manager.py", line 17, in
from nbformat import sign, validate as validate_nb, ValidationError
File "D:\Anaconda3\lib\site-packages\nbformat\__init__.py", line 32, in
from .validator import validate, ValidationError
File "D:\Anaconda3\lib\site-packages\nbformat\validator.py", line 12, in
from .json_compat import get_current_validator, ValidationError
File "D:\Anaconda3\lib\site-packages\nbformat\json_compat.py", line 10, in
import jsonschema
File "D:\Anaconda3\lib\site-packages\jsonschema\__init__.py", line 33, in
import importlib_metadata as metadata
ModuleNotFoundError: No module named 'importlib_metadata'
Dose anyone know how to solve this problem?
I have created virtualenv for python3 using:
virtualenv -p $(which python3) ENV
Then activate the source
source /Users/myusername/ENV/bin/activate
Install the apache-airflow:
pip install apache-airflow
then which airflow yields /Users/myusername/ENV/bin/airflow
But when I try to initdb using:
airflow initdb
I get below error:
{db.py:350} INFO - Creating tables
INFO [alembic.runtime.migration] Context impl SQLiteImpl.
INFO [alembic.runtime.migration] Will assume non-transactional DDL.
WARNI [airflow.utils.log.logging_mixin.LoggingMixin] cryptography not found - values will not be stored encrypted.
ERROR [airflow.models.DagBag] Failed to import: /Library/Python/2.7/site-packages/airflow/example_dags/example_http_operator.py
Traceback (most recent call last):
File "/Library/Python/2.7/site-packages/airflow/models/__init__.py", line 413, in process_file
m = imp.load_source(mod_name, filepath)
File "/Library/Python/2.7/site-packages/airflow/example_dags/example_http_operator.py", line 27, in <module>
from airflow.operators.http_operator import SimpleHttpOperator
File "/Library/Python/2.7/site-packages/airflow/operators/http_operator.py", line 21, in <module>
from airflow.hooks.http_hook import HttpHook
File "/Library/Python/2.7/site-packages/airflow/hooks/http_hook.py", line 23, in <module>
import tenacity
File "/Library/Python/2.7/site-packages/tenacity/__init__.py", line 375, in <module>
from tenacity.tornadoweb import TornadoRetrying
File "/Library/Python/2.7/site-packages/tenacity/tornadoweb.py", line 24, in <module>
from tornado import gen
File "/Library/Python/2.7/site-packages/tornado-6.0.3-py2.7-macosx-10.14-intel.egg/tornado/gen.py", line 126
def _value_from_stopiteration(e: Union[StopIteration, "Return"]) -> Any:
^
SyntaxError: invalid syntax
Done.
(ENV) ---------------------------------------------------------
Seems like example scripts use python 2.7 and it can't recognize the function definition syntax.
Does apache-airflow package need to be fixed by the next release or I can do something to fix this?
I tried fixing this:
Use python2.7 instead of python3
then install airflow on default python 2.7 enabled on mac but this throws other errors like package "six" is not compatible.
You need to turn off the example DAGs to be loaded in config file to solve this problem.
Anyway, it seems weird that airflow uses 2.7 Python when you told that it is installed into Python 3 virtual environment.