I am trying to configure remote logging with Azure blob.
Airflow version: 1.10.2
Python: 3.6.5
Ubuntu: 18.04
Following are the step I did:
In $AIRFLOW_HOME/config/log_config.py, I have put REMOTE_BASE_LOG_FOLDER = 'wasb-airflow-logs' (This is a folder inside the container (container name: airflow-logs))
Empty init.py is in $AIRFLOW_HOME/config/
$AIRFLOW_HOME/config/ is added in $PYTHONPATH
Renamed DEFAULT_LOGGING_CONFIG to LOGGING CONFIG everywhere in $AIRFLOW_HOME/config/log_config.py
User defined in Airflow blob connection has read/write access to REMOTE_BASE_LOG_FOLDER
$AIRFLOW_HOME/airflow.cfg it has remote_logging = True
logging_config_class = log_config.LOGGING_CONFIG
remote_log_conn_id =
Following is the error:
Unable to load the config, contains a configuration error.
Traceback (most recent call last):
File "/home/gsingh/anaconda3/lib/python3.6/logging/config.py", line 382, in resolve
found = getattr(found, frag)
AttributeError: module 'airflow.utils.log' has no attribute 'wasb_task_handler'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/gsingh/anaconda3/lib/python3.6/logging/config.py", line 384, in resolve
self.importer(used)
File "/home/gsingh/venv/lib/python3.6/site-packages/airflow/utils/log/wasb_task_handler.py", line 23, in <module>
from airflow.contrib.hooks.wasb_hook import WasbHook
File "/home/gsingh/venv/lib/python3.6/site-packages/airflow/contrib/hooks/wasb_hook.py", line 22, in <module>
from airflow.hooks.base_hook import BaseHook
File "/home/gsingh/venv/lib/python3.6/site-packages/airflow/hooks/base_hook.py", line 28, in <module>
from airflow.models import Connection
File "/home/gsingh/venv/lib/python3.6/site-packages/airflow/models.py", line 86, in <module>
from airflow.utils.dag_processing import list_py_file_paths
File "/home/gsingh/venv/lib/python3.6/site-packages/airflow/utils/dag_processing.py", line 49, in <module>
from airflow.settings import logging_class_path
ImportError: cannot import name 'logging_class_path'
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/home/gsingh/anaconda3/lib/python3.6/logging/config.py", line 558, in configure
handler = self.configure_handler(handlers[name])
File "/home/gsingh/anaconda3/lib/python3.6/logging/config.py", line 708, in configure_handler
klass = self.resolve(cname)
File "/home/gsingh/anaconda3/lib/python3.6/logging/config.py", line 391, in resolve
raise v
File "/home/gsingh/anaconda3/lib/python3.6/logging/config.py", line 384, in resolve
self.importer(used)
File "/home/gsingh/venv/lib/python3.6/site-packages/airflow/utils/log/wasb_task_handler.py", line 23, in <module>
from airflow.contrib.hooks.wasb_hook import WasbHook
File "/home/gsingh/venv/lib/python3.6/site-packages/airflow/contrib/hooks/wasb_hook.py", line 22, in <module>
from airflow.hooks.base_hook import BaseHook
File "/home/gsingh/venv/lib/python3.6/site-packages/airflow/hooks/base_hook.py", line 28, in <module>
from airflow.models import Connection
File "/home/gsingh/venv/lib/python3.6/site-packages/airflow/models.py", line 86, in <module>
from airflow.utils.dag_processing import list_py_file_paths
File "/home/gsingh/venv/lib/python3.6/site-packages/airflow/utils/dag_processing.py", line 49, in <module>
from airflow.settings import logging_class_path
ValueError: Cannot resolve 'airflow.utils.log.wasb_task_handler.WasbTaskHandler': cannot import name 'logging_class_path'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/gsingh/venv/bin/airflow", line 21, in <module>
from airflow import configuration
File "/home/gsingh/venv/lib/python3.6/site-packages/airflow/__init__.py", line 36, in <module>
from airflow import settings, configuration as conf
File "/home/gsingh/venv/lib/python3.6/site-packages/airflow/settings.py", line 262, in <module>
logging_class_path = configure_logging()
File "/home/gsingh/venv/lib/python3.6/site-packages/airflow/logging_config.py", line 73, in configure_logging
raise e
File "/home/gsingh/venv/lib/python3.6/site-packages/airflow/logging_config.py", line 68, in configure_logging
dictConfig(logging_config)
File "/home/gsingh/anaconda3/lib/python3.6/logging/config.py", line 795, in dictConfig
dictConfigClass(config).configure()
File "/home/gsingh/anaconda3/lib/python3.6/logging/config.py", line 566, in configure
'%r: %s' % (name, e))
ValueError: Unable to configure handler 'processor': Cannot resolve 'airflow.utils.log.wasb_task_handler.WasbTaskHandler': cannot import name 'logging_class_path'
I am not sure which configuration I am missing. Has anyone faced the same issue?
You need to install the azure package.
pip install 'apache-airflow[azure_blob_storage,azure_data_lake,azure_cosmos,azure_container_instances]
As per updating.md
This now should be installed with
pip install apache-airflow[azure]
But this didn't work for me.
sudo chown 50000:0 dags logs plugins in my case.
I tried to run official docker-compose.yml with all these containers (which are dependent on these 3 volume forwards) or simply wrap airflow standalone into a single container for a debug purpose. Turned out volumes were created with root ownerships instead of airflows.
I had the same error however if I scrolled up higher I could see that there was another exception thrown before the ValueError. Which was a PermissionError.
PermissionError: [Errno 13] Permission denied: '/usr/local/airflow/logs/scheduler'
The reason I got that error is because I didn't create the initial 3 folders (dags, logs, plugins) before running airflow docker container. So docker seems to have created then automatically but the permissions were wrong.
Steps to fix:
Stop current container
docker-compose down --volumes --remove-orphans
Delete folders dags, logs, plugins
Just in case Destroy the images and volumes already created (in Docker Desktop)
Create folders again from command line
mkdir logs dags plugins
run airflow docker again
docker-compose up airflow-init
docker-compose up
Related
I installed on a fresh version of airflow using Python 3.7.4 on Mac OS Catalina 10.15.1.
By running:
$ pip install 'apache-airflow[s3,postgres,celery,hdfs,jdbc,hive,slack,crypto,redis]==1.10.4'
When I run:
$ airflow initidb
I get the following error message:
Traceback (most recent call last):
File "/Users/user/opt/anaconda3/bin/airflow", line 21, in <module>
from airflow import configuration
File "/Users/user/opt/anaconda3/lib/python3.7/site-packages/airflow/__init__.py", line 31, in <module>
from airflow.utils.log.logging_mixin import LoggingMixin
File "/Users/user/opt/anaconda3/lib/python3.7/site-packages/airflow/utils/__init__.py", line 24, in <module>
from .decorators import apply_defaults as _apply_defaults
File "/Users/user/opt/anaconda3/lib/python3.7/site-packages/airflow/utils/decorators.py", line 34, in <module>
from airflow import settings
File "/Users/user/opt/anaconda3/lib/python3.7/site-packages/airflow/settings.py", line 36, in <module>
from airflow.configuration import conf, AIRFLOW_HOME, WEBSERVER_CONFIG # NOQA F401
File "/Users/user/opt/anaconda3/lib/python3.7/site-packages/airflow/configuration.py", line 29, in <module>
from future import standard_library
ValueError: source code string cannot contain null bytes
I'm wondering if anyone has any suggestions.
Thanks.
I think I figured it out. The future module I had was corrupt, so I just re-installed things and it seems to work now.
I read where running installing pyqt5 or pip install pyqt5 would correct this error. Did not work for my enviornment. Any suggestions?
Traceback (most recent call last):
File "C:\Users\Paula\Anaconda3\lib\site-packages\notebook\services\sessions\sessionmanager.py", line 10, in
import sqlite3
File "C:\Users\Paula\Anaconda3\lib\sqlite3\__init__.py", line 23, in
from sqlite3.dbapi2 import *
File "C:\Users\Paula\Anaconda3\lib\sqlite3\dbapi2.py", line 27, in
from _sqlite3 import *
ImportError: DLL load failed: The specified procedure could not be found.
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "C:\Users\Paula\Anaconda3\Scripts\jupyter-notebook-script.py", line 6, in
from notebook.notebookapp import main
File "C:\Users\Paula\Anaconda3\lib\site-packages\notebook\notebookapp.py", line 86, in
from .services.sessions.sessionmanager import SessionManager
File "C:\Users\Paula\Anaconda3\lib\site-packages\notebook\services\sessions\sessionmanager.py", line 13, in
from pysqlite2 import dbapi2 as sqlite3
ModuleNotFoundError: No module named 'pysqlite2'
I've just reinstalled rpy2 with Python3.5 and I am having difficulty figuring out why the module cannot find the "R-Home". I have installed R on my computer already. Should I try reinstalling R or another method? Some help would be much appreciated here!
import rpy2.rinterface
Traceback (most recent call last):
File "<ipython-input-7-676c977874d9>", line 1, in <module>
import rpy2.rinterface
File "/Users/JasonDucker/anaconda/lib/python3.5/site-packages/rpy2/rinterface/__init__.py", line 16, in <module>
tmp = subprocess.check_output(("R", "RHOME"), universal_newlines=True)
File "/Users/JasonDucker/anaconda/lib/python3.5/subprocess.py", line 629, in check_output
**kwargs).stdout
File "/Users/JasonDucker/anaconda/lib/python3.5/subprocess.py", line 696, in run
with Popen(*popenargs, **kwargs) as process:
File "/Users/JasonDucker/anaconda/lib/python3.5/subprocess.py", line 950, in __init__
restore_signals, start_new_session)
File "/Users/JasonDucker/anaconda/lib/python3.5/subprocess.py", line 1544, in _execute_child
raise child_exception_type(errno_num, err_msg)
FileNotFoundError: [Errno 2] No such file or directory: 'R'
This means that there is no executable in the PATH when python is calling "R RHOME" in a subprocess.
To overcome this add the path to R to your PATH. For example:
export PATH=/usr/local/some/place/bin:${PATH}
I've just installed a new plone 4.3 instance in a folder which already contained 2 Plone 4.3 instance, which both work fine. When I try to run buildout I get this error:
Traceback (most recent call last):
File "bin/buildout", line 17, in <module>
import zc.buildout.buildout
File "/usr/local/Plone4.3/buildout-cache/eggs/zc.buildout-1.7.1-py2.7.egg/zc/buildout/buildout.py", line 40, in <module>
import zc.buildout.download
File "/usr/local/Plone4.3/buildout-cache/eggs/zc.buildout-1.7.1-py2.7.egg/zc/buildout/download.py", line 20, in <module>
from zc.buildout.easy_install import realpath
File "/usr/local/Plone4.3/buildout-cache/eggs/zc.buildout-1.7.1-py2.7.egg/zc/buildout/easy_install.py", line 31, in <module>
import setuptools.package_index
File "/usr/local/Plone4.3/Python-2.7/lib/python2.7/site-packages/distribute-0.6.35-py2.7.egg/setuptools/package_index.py", line 158, in <module>
sys.version[:3], require('distribute')[0].version
File "/usr/local/Plone4.3/buildout-cache/eggs/setuptools-0.6c11-py2.7.egg/pkg_resources.py", line 666, in require
continue # try the next older version of project
File "/usr/local/Plone4.3/buildout-cache/eggs/setuptools-0.6c11-py2.7.egg/pkg_resources.py", line 565, in resolve
pkg_resources.DistributionNotFound: distribute
Google has failed me. Can anyone tell me how to resolve this error?
I updated appengine and now manage.py is broken, this is what I get:
Traceback (most recent call last):
File "manage.py", line 18, in
InstallAppengineHelperForDjango()
File "\workspace\Project\src\appengine_django__init__.py", line
545, in InstallAppengineHelperForDjango
InstallDjangoModuleReplacements()
File "\workspace\Project\src\appengine_django__init__.py", line
262, in InstallDjangoModuleReplacements
import django.db
File "\workspace\Project\src\django.zip\django\db__init__.py", l
ine 39, in
File "\workspace\Project\src\appengine_django\db\base.py", line 1
17, in init
self._setup_stubs()
File "\workspace\Project\src\appengine_django\db\base.py", line 1
28, in _setup_stubs
from google.appengine.tools import dev_appserver_main
File "C:\Program Files (x86)\Google\google_appengine\google\appengine\tools\de
v_appserver_main.py", line 90, in
from google.appengine.tools import appcfg
File "C:\Program Files (x86)\Google\google_appengine\google\appengine\tools\ap
pcfg.py", line 59, in
from google.appengine.tools import appengine_rpc
File "C:\Program Files (x86)\Google\google_appengine\google\appengine\tools\ap
pengine_rpc.py", line 24, in
import fancy_urllib
ImportError: No module named fancy_urllib
Any help is appreciated.
I am getting the same with the recent App Engine SDK 1.3.6 release + Google App Engine Helper for Django. Google is aware of the issue and a ticket has been opened here: http://code.google.com/p/google-app-engine-django/issues/detail?id=178; please star it to escalate.
A couple options:
wait for them to issue a patch
rollback to 1.3.5. I had to uninstall 1.3.6 before reinstalling 1.3.5.