Airflow v1.9.0 docker deployment based on puckel just broke for me with this error:
Traceback (most recent call last):
File "/usr/local/bin/airflow", line 27, in <module>
args.func(args)
File "/usr/local/lib/python2.7/dist-packages/airflow/bin/cli.py", line 891, in worker
worker.run(**options)
File "/usr/local/lib/python2.7/dist-packages/celery/bin/worker.py", line 255, in run
**kwargs)
File "/usr/local/lib/python2.7/dist-packages/celery/worker/worker.py", line 99, in __init__
self.setup_instance(**self.prepare_args(**kwargs))
File "/usr/local/lib/python2.7/dist-packages/celery/worker/worker.py", line 122, in setup_instance
self.should_use_eventloop() if use_eventloop is None
File "/usr/local/lib/python2.7/dist-packages/celery/worker/worker.py", line 241, in should_use_eventloop
self._conninfo.transport.implements.async and
File "/usr/local/lib/python2.7/dist-packages/kombu/transport/base.py", line 125, in __getattr__
raise AttributeError(key)
AttributeError: async
The Cobman solution works, just FYI you can fix it too upgrading the Celery version (as they recommend in their repository)
&& pip install celery[redis]==4.1.1
It is related to Celery AttributeError: async error error just reported where kombu was updated from 4.1.0 to 4.2.0. I fixed it by switching the install order per below:
&& pip install kombu==4.1.0 \
&& pip install celery[redis]==4.0.2 \
&& pip install apache-airflow[crypto,celery,postgres,hive,jdbc,mysql,s3]==$AIRFLOW_VERSION \
Seems like kombu needs to pinned to this version in source...
Related
I have just installed Anaconda and run Jupyter Notebook through command line by using jupyter notebook command. However, I am facing an issue. On the top right, I see Kernel Error.
When I click on Kernel Error, I see these lines:
Traceback (most recent call last):
File "C:\Users\Shawn\AppData\Roaming\Python\Python37\site-packages\tornado\web.py", line 1699, in _execute
result = await result
File "C:\Users\Shawn\AppData\Roaming\Python\Python37\site-packages\tornado\gen.py", line 742, in run
yielded = self.gen.throw(*exc_info) # type: ignore
File "C:\Users\Shawn\AppData\Roaming\Python\Python37\site-packages\notebook\services\sessions\handlers.py", line 72, in post
type=mtype))
File "C:\Users\Shawn\AppData\Roaming\Python\Python37\site-packages\tornado\gen.py", line 735, in run
value = future.result()
File "C:\Users\Shawn\AppData\Roaming\Python\Python37\site-packages\tornado\gen.py", line 742, in run
yielded = self.gen.throw(*exc_info) # type: ignore
File "C:\Users\Shawn\AppData\Roaming\Python\Python37\site-packages\notebook\services\sessions\sessionmanager.py", line 88, in create_session
kernel_id = yield self.start_kernel_for_session(session_id, path, name, type, kernel_name)
File "C:\Users\Shawn\AppData\Roaming\Python\Python37\site-packages\tornado\gen.py", line 735, in run
value = future.result()
File "C:\Users\Shawn\AppData\Roaming\Python\Python37\site-packages\tornado\gen.py", line 742, in run
yielded = self.gen.throw(*exc_info) # type: ignore
File "C:\Users\Shawn\AppData\Roaming\Python\Python37\site-packages\notebook\services\sessions\sessionmanager.py", line 101, in start_kernel_for_session
self.kernel_manager.start_kernel(path=kernel_path, kernel_name=kernel_name)
File "C:\Users\Shawn\AppData\Roaming\Python\Python37\site-packages\tornado\gen.py", line 735, in run
value = future.result()
File "C:\Users\Shawn\AppData\Roaming\Python\Python37\site-packages\tornado\gen.py", line 209, in wrapper
yielded = next(result)
File "C:\Users\Shawn\AppData\Roaming\Python\Python37\site-packages\notebook\services\kernels\kernelmanager.py", line 168, in start_kernel
super(MappingKernelManager, self).start_kernel(**kwargs)
File "C:\ProgramData\Anaconda3\lib\site-packages\jupyter_client\multikernelmanager.py", line 158, in start_kernel
km.start_kernel(**kwargs)
File "C:\ProgramData\Anaconda3\lib\site-packages\jupyter_client\manager.py", line 301, in start_kernel
kernel_cmd, kw = self.pre_start_kernel(**kw)
File "C:\ProgramData\Anaconda3\lib\site-packages\jupyter_client\manager.py", line 248, in pre_start_kernel
self.write_connection_file()
File "C:\ProgramData\Anaconda3\lib\site-packages\jupyter_client\connect.py", line 474, in write_connection_file
kernel_name=self.kernel_name
File "C:\ProgramData\Anaconda3\lib\site-packages\jupyter_client\connect.py", line 138, in write_connection_file
with secure_write(fname) as f:
File "C:\ProgramData\Anaconda3\lib\contextlib.py", line 112, in __enter__
return next(self.gen)
File "C:\Users\Shawn\AppData\Roaming\Python\Python37\site-packages\jupyter_core\paths.py", line 424, in secure_write
win32_restrict_file_to_user(fname)
File "C:\Users\Shawn\AppData\Roaming\Python\Python37\site-packages\jupyter_core\paths.py", line 359, in win32_restrict_file_to_user
import win32api
ImportError: DLL load failed: %1 is not a valid Win32 application.
I can't run any Python code on Jupyter Notebook, as I am continuing to get that error.
I have gone through numerous threads on Stack Overflow for several hours. But I can't fix the issue.
Please, help me to solve the issue. I am doing a course on Data Scince with Python. I can't move forward without fixing this issue.
Update 1:
I have visited this Github thread and tried using the following command:
conda install -c conda-forge pywin32
conda install -c anaconda jupyter_client
conda install -c conda-forge jupyter_core
pip install --upgrade pywin32==224
However, the issue is not fixed. Instead, I am getting a new error,
ImportError: DLL load failed: The specified procedure could not be found.
This is really frustrating!
try this way, i fixed my jupyter.
same error site-packages\tornado
command
jupyter kernelspec list
from the shown result
Available kernels:
python3 D:\Programs\Anaconda3\share\jupyter\kernels\python3
open the folder, and edit kernel.json
{
"argv": [
"D:\\Programs\\Anaconda3\\python.exe", //change to your correct path
"-m",
"ipykernel_launcher",
"-f",
"{connection_file}"
],
"display_name": "Python 3",
"language": "python"
}
restart jupyter, and it will be ok.
Did you try opening the anaconda navigator from terminal and then create an environment their and install the jupyter notebook in that environment and then open the jupyter notebook from the navigator
Alright. I have managed to fix the issue. I have to write these commands:
conda install -c conda-forge pywin32
conda install -c anaconda jupyter_client
conda install -c conda-forge jupyter_core
pip install --upgrade pywin32==224
Then I got an error.
import win32api pywin32
ImportError: DLL load failed: The specified procedure could not be found.
I solve the issue by writing this command that I got from this github thread:
conda install pywin32
Now, the issue has been completely fixed!
Can anyone support me this case?
I install Robotframework to a machine (win10 64b). The versions in my machine:
Python3.7.2 (32b)
pywin32==224 (32b)
robotframework==3.1.1
robotframework-autoitlibrary==1.2.3
robotframework-ride==1.7.3.1
robotframework-seleniumlibrary==3.3.1
wxPython==4.0.4 (32b)
Import AutoItLibary, it is always red, Ride error log shows:
Traceback (most recent call last):
Initializing test library 'AutoItLibrary' with no arguments failed: RobotNotRunningError: Cannot access execution context
Traceback (most recent call last):
File "c:\python37-32\lib\site-packages\robotide\lib\robot\running\testlibraries.py", line 167, in _get_instance
return libcode(*self.positional_args, **dict(self.named_args))
File "c:\python37-32\lib\site-packages\AutoItLibrary\__init__.py", line 88, in __init__
self._OutputDir = self._get_log_dir()
File "c:\python37-32\lib\site-packages\AutoItLibrary\__init__.py", line 182, in _get_log_dir
variables = BuiltIn().get_variables()
File "c:\python37-32\lib\site-packages\robot\libraries\BuiltIn.py", line 1293, in get_variables
return self._variables.as_dict(decoration=is_falsy(no_decoration))
File "c:\python37-32\lib\site-packages\robot\libraries\BuiltIn.py", line 75, in _variables
return self._namespace.variables
File "c:\python37-32\lib\site-packages\robot\libraries\BuiltIn.py", line 71, in _namespace
return self._get_context().namespace
File "c:\python37-32\lib\site-packages\robot\libraries\BuiltIn.py", line 66, in _get_context
raise RobotNotRunningError('Cannot access execution context')
File "c:\python37-32\lib\site-packages\robotide\spec\librarymanager.py", line 87, in _fetch_keywords
return get_import_result(path, library_args)
File "c:\python37-32\lib\site-packages\robotide\spec\libraryfetcher.py", line 24, in get_import_result
lib = robotapi.TestLibrary(path, args)
File "c:\python37-32\lib\site-packages\robotide\lib\robot\running\testlibraries.py", line 54, in TestLibrary
lib.create_handlers()
File "c:\python37-32\lib\site-packages\robotide\lib\robot\running\testlibraries.py", line 100, in create_handlers
self._create_handlers(self.get_instance())
File "c:\python37-32\lib\site-packages\robotide\lib\robot\running\testlibraries.py", line 159, in get_instance
self._libinst = self._get_instance(self._libcode)
File "c:\python37-32\lib\site-packages\robotide\lib\robot\running\testlibraries.py", line 169, in _get_instance
self._raise_creating_instance_failed()
File "c:\python37-32\lib\site-packages\robotide\lib\robot\running\testlibraries.py", line 314, in _raise_creating_instance_failed
% (self.name, args_text, msg, details))
20190326 17:31:22.876 [INFO]: Found Robot Framework version 3.1.1 from c:\python37-32\lib\site-packages\robot.
20190326 17:31:22.876 [INFO]: Started RIDE 1.7.3.1 using python version 3.7.2 with wx version 4.0.4 in win32. ```
I had the same problem with the robotframework-autoitlibrary.
The way I found to fix and use autoit in RF was to uninstall the "robotframework-autoitlibrary" and install the "autoitlibrary" found here: https://code.google.com/archive/p/robotframework-autoitlibrary/ with python setup.py install as Admin.
Initializing error is now gone.
AutoItLibrary is not built in with robot framework you need to import the library.
pip install robotframework-autoitlibrary
I've completed a manual installation of Eucalpytus 4.4.4 but when I try to use a web browser to reach the eucaconsole (running on the same host as CLC/UFS) I get a 502 Bad Gateway Error.
I'm focusing on this error in the eucaconsole.log What does it mean and how can I update WebOb?
pkg_resources.VersionConflict: (WebOb 1.2.3 (/usr/lib/python2.7/site-packages), Requirement.parse('WebOb>=1.3.1'))
Eucaconsole_startup.log:
Traceback (most recent call last):
File "/bin/eucaconsole", line 106, in <module>
daemonize(start_console)
File "/bin/eucaconsole", line 61, in daemonize
func()
File "/bin/eucaconsole", line 73, in start_console
load_entry_point('pyramid', 'console_scripts', 'pserve')(args)
File "/usr/lib/python2.7/site-packages/pkg_resources.py", line 378, in load_entry_point
return get_distribution(dist).load_entry_point(group, name)
File "/usr/lib/python2.7/site-packages/pkg_resources.py", line 2566, in load_entry_point
return ep.load()
File "/usr/lib/python2.7/site-packages/pkg_resources.py", line 2259, in load
if require: self.require(env, installer)
File "/usr/lib/python2.7/site-packages/pkg_resources.py", line 2272, in require
working_set.resolve(self.dist.requires(self.extras),env,installer)))
File "/usr/lib/python2.7/site-packages/pkg_resources.py", line 630, in resolve
raise VersionConflict(dist,req) # XXX put more info here
pkg_resources.VersionConflict: (WebOb 1.2.3 (/usr/lib/python2.7/site-packages), Requirement.parse('WebOb>=1.3.1'))
I've edited this post to remove info and focus on the clear WebOb version error.
This issue was solved below. Adding a note that the downlevel python-webob version is actually a requirement of the midonet client installation. So if you expect to run VPCMIDO and have the midonet gateway on your CLC you'll have to run your eucaconsole elsewhere.
This was resolved by removing an older python-webob package to ensure that the newer python-webob1.4 package from epel was used.
Related issue in github:
https://github.com/Corymbia/eucalyptus/issues/124
Selinux problem. Run the following in the your CLC/UFS machine
setsebool -P httpd_can_network_connect 1
Its better to flush your iptables during installation.
I'm trying to setup piping to output a pipfile for my employer. I'm totally new to this, and my system knowledge is lacking. Please can someone help me debug this error?
When I run :
pipenv install -e
I get the following error :
(py3) ➜ ~ git:(master) ✗ pipenv install -e
Creating a virtualenv for this project...
Pipfile: /Users/davidmellor/Pipfile
Using /Users/davidmellor/anaconda/envs/py3/bin/python (3.6.2) to create virtualenv...
⠋Already using interpreter /Users/davidmellor/anaconda/envs/py3/bin/python
Using base prefix '/Users/davidmellor/anaconda/envs/py3'
New python executable in /Users/davidmellor/.local/share/virtualenvs/davidmellor-1NR562gx/bin/python
dyld: Library not loaded: #rpath/libpython3.6m.dylib
Referenced from: /Users/davidmellor/.local/share/virtualenvs/davidmellor-1NR562gx/bin/python
Reason: image not found
ERROR: The executable /Users/davidmellor/.local/share/virtualenvs/davidmellor-1NR562gx/bin/python is not functioning
ERROR: It thinks sys.prefix is '/Users/davidmellor' (should be '/Users/davidmellor/.local/share/virtualenvs/davidmellor-1NR562gx')
ERROR: virtualenv is not compatible with this system or executable
Virtualenv location:
Creating a Pipfile for this project...
Traceback (most recent call last):
File "/Users/davidmellor/anaconda/envs/py3/bin/pipenv", line 11, in <module>
sys.exit(cli())
File "/Users/davidmellor/.local/lib/python3.6/site-packages/pipenv/vendor/click/core.py", line 722, in __call__
return self.main(*args, **kwargs)
File "/Users/davidmellor/.local/lib/python3.6/site-packages/pipenv/vendor/click/core.py", line 697, in main
rv = self.invoke(ctx)
File "/Users/davidmellor/.local/lib/python3.6/site-packages/pipenv/vendor/click/core.py", line 1066, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/Users/davidmellor/.local/lib/python3.6/site-packages/pipenv/vendor/click/core.py", line 895, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "/Users/davidmellor/.local/lib/python3.6/site-packages/pipenv/vendor/click/core.py", line 535, in invoke
return callback(*args, **kwargs)
File "/Users/davidmellor/.local/lib/python3.6/site-packages/pipenv/cli.py", line 435, in install
selective_upgrade=selective_upgrade,
File "/Users/davidmellor/.local/lib/python3.6/site-packages/pipenv/core.py", line 1759, in do_install
pypi_mirror=pypi_mirror,
File "/Users/davidmellor/.local/lib/python3.6/site-packages/pipenv/core.py", line 639, in ensure_project
ensure_pipfile(validate=validate, skip_requirements=skip_requirements, system=system)
File "/Users/davidmellor/.local/lib/python3.6/site-packages/pipenv/core.py", line 288, in ensure_pipfile
project.create_pipfile(python=python)
File "/Users/davidmellor/.local/lib/python3.6/site-packages/pipenv/project.py", line 607, in create_pipfile
required_python = self.which('python')
File "/Users/davidmellor/.local/lib/python3.6/site-packages/pipenv/core.py", line 115, in which
p = os.path.join(location, 'bin', command)
File "/Users/davidmellor/anaconda/envs/py3/lib/python3.6/posixpath.py", line 78, in join
a = os.fspath(a)
TypeError: expected str, bytes or os.PathLike object, not NoneType
I've tried to modify my PATH within my .bashrc file as follows :
PYTHON_BIN_PATH="$/Users/davidmellor/.local/bin"
PATH="$PATH:$PYTHON_BIN_PATH"
export PATH=$HOME/.composer/vendor/bin
export DJANGO_SETTINGS_MODULE=blog.settings
But still having no luck.
Thank you so much.
Try using 'pyenv' package with pipenv.
Install 'pyenv' package by using brew install pyenv (if you don't have it).
Install python 3.6.2 using pyenv install 3.6.2. In your case you are using anaconda, so you can also install any anaconda version you would like to use.
Use pyenv install --list to see the list of python available versions
Then, export new installed python version to PATH
export PATH=${PYENV_PYTHON_VERSIONS_HOME}/3.6.2/bin
Now in 'Piplock' specify the same version.
[requires]
python_version = "3.6.2"
Finally, run remove previous virtualenv and rebuild again.
pipenv --rm
pipenv install --dev.
I used the book "Deep Learning with R" since one month now, and it enables me to make my first neural networks.
I am using Ubuntu. Until 2 days ago, everything was OK and worked fine. But two days ago I updated my Ubuntu to Ubuntu 18.02. Since then, my R code is not working anymore.
I have re-done what is recommended in the book (and what has worked one month ago):
$ sudo apt-get update
$ sudo apt-get upgrade
$ sudo apt-get install python-pip python-dev
$ sudo apt-get install build-essential cmake git unzip pkg-config libopenblas-dev liblapack-dev
I had no error.
then in R:
install.packages("keras")
library(keras)
install_keras()
This last command is supposed to install the core Keras library along with its dependencies in a Python virtual environment using TensorFlow.
But I obtained the following error that I really do not understand:
> install_keras()
Using existing virtualenv at ~/.virtualenvs/r-tensorflow
Upgrading pip ...
Traceback (most recent call last):
File "/home/baragatt/.virtualenvs/r-tensorflow/bin/pip", line 7, in <module>
from pip._internal import main
File "/home/baragatt/.virtualenvs/r-tensorflow/local/lib/python2.7/site-packages/pip/_internal/__init__.py", line 5, in <module>
import logging
File "/usr/lib/python2.7/logging/__init__.py", line 26, in <module>
import sys, os, time, cStringIO, traceback, warnings, weakref, collections
File "/usr/lib/python2.7/weakref.py", line 14, in <module>
from _weakref import (
ImportError: cannot import name _remove_dead_weakref
Erreur : Error 1 occurred installing TensorFlow
I have re-installed R, python, tensorflow, but I always have the same error. I do not understand this error. Maybe this is a problem with the virtualenv?
Can someone help me please? It is so frustrating, because two days ago my code was running, and now impossible to work...
I am working with Ubuntu 18.02, at the installed versions are python 2.7.15~rc1-1, R-3.4.4 and tensorflow-1.10.0.
Thanks a lot for this post. I do not really understand what the commands in this post are supposed to fix. But I have done the following:
cd /home/baragatt/.virtualenvs/r-tensorflow/
Then, as proposed in the post:
virtualenv . --system-site-packages
I obtained the following messages:
Running virtualenv with interpreter /usr/bin/python2
New python executable in /home/baragatt/.virtualenvs/r-tensorflow/bin/python2
Not overwriting existing python script /home/baragatt/.virtualenvs/r-tensorflow/bin/python (you must use /home/baragatt/.virtualenvs/r-tensorflow/bin/python2)
Traceback (most recent call last):
File "/usr/lib/python3/dist-packages/virtualenv.py", line 2375, in <module>
main()
File "/usr/lib/python3/dist-packages/virtualenv.py", line 724, in main
symlink=options.symlink)
File "/usr/lib/python3/dist-packages/virtualenv.py", line 946, in create_environment
site_packages=site_packages, clear=clear, symlink=symlink))
File "/usr/lib/python3/dist-packages/virtualenv.py", line 1417, in install_python
os.symlink(py_executable_base, full_pth)
OSError: [Errno 17] File exists
I also tried:
virtualenv -p /usr/bin/python2.7 .
And I obtained:
Running virtualenv with interpreter /usr/bin/python2.7
New python executable in /home/baragatt/.virtualenvs/r-tensorflow/bin/python2.7
Traceback (most recent call last):
File "/usr/lib/python3/dist-packages/virtualenv.py", line 2375, in <module>
main()
File "/usr/lib/python3/dist-packages/virtualenv.py", line 724, in main
symlink=options.symlink)
File "/usr/lib/python3/dist-packages/virtualenv.py", line 946, in create_environment
site_packages=site_packages, clear=clear, symlink=symlink))
File "/usr/lib/python3/dist-packages/virtualenv.py", line 1278, in install_python
shutil.copyfile(executable, py_executable)
File "/usr/lib/python2.7/shutil.py", line 97, in copyfile
with open(dst, 'wb') as fdst:
IOError: [Errno 40] Too many levels of symbolic links: '/home/baragatt/.virtualenvs/r-tensorflow/bin/python2.7'
I finally find a solution, by looking at different forums.
I thought that the problem should be because of the virtualenvironment that should be created when doing the following command in R.
install_keras()
Hence, I deleted the virtual environment(s) by deleting the directory in which these environements are located (I imagine).
cd ~/.virtualenvs
rm -r r-tensorflow/
Then I have tried the following commands in R
install.packages("keras")
library(keras)
install_keras()
And it works! Honestly, I still do not understand what was the problem that occured after my Ubuntu update.