I am trying to deploy an Airfow DAG to MWAA.
My requirements.txt:
apache-airflow[amazon] == 3.2.0
I import EcsOperator like this:
from airflow.contrib.operators.ecs_operator import EcsOperator
However, I get this error:
Broken DAG: [/usr/local/airflow/dags/mydag.py] Traceback (most recent call last):
File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
File "/usr/local/airflow/dags/mydag.py", line 4, in <module>
from airflow.contrib.operators.ecs_operator import EcsOperator
ImportError: cannot import name 'EcsOperator' from 'airflow.contrib.operators.ecs_operator' (/usr/local/lib/python3.7/site-packages/airflow/contrib/operators/ecs_operator.py)
What am I doing wrong here?
What am I doing wrong here?
You might be referencing a different version (1.10.12?) of the Airflow documentation.
airflow.contrib.operators.ecs_operator (1.10.12)
The documentation for 3.2.0 is here. You can import the EcsOperator like this:
from airflow.providers.amazon.aws.operators.ecs import EcsOperator
airflow.providers.amazon.aws.operators.ecs (3.2.0)
The correct requirements.txt:
(empty)
And the correct import:
from airflow.providers.amazon.aws.operators.ecs import ECSOperator
Note the casing!
There are several issues here so I'll compile a detailed answer since privious answers didn't cover all of them.
First, the updated import path (provider release 3.2.0) is:
from airflow.providers.amazon.aws.operators.ecs import EcsOperator
The reason this doesn't work for you is because you install the provider with extras as:
apache-airflow[amazon]
as explained in the provider extra docs when installing provider in that manner you get the provider version which was released at the time of the Airflow version that you are using. Thus you are not guaranteed to get the updated provider version. So in case you are using Airflow 2.2.4 (latest at the time of writing this answer) you will get Amazon provider version 3.0.0 which is not the most recent one.
To get updated provider you should install it as:
pip install apache-airflow-providers-amazon
if you like to pick a specific version then:
pip install apache-airflow-providers-amazon==3.2.0
Please note that you should always install from constraint files provided by Airflow. Example:
pip install "apache-airflow-providers-amazon" --constraint "https://raw.githubusercontent.com/apache/airflow/constraints-main/constraints-3.7.txt"
Note that the provider is referring to constraints-main which constantly updated and not to constraints-2.2.4 or any other specific Airflow version.
You can read more about it in the doc about Installation and upgrading of Airflow providers separately.
Related
Trying to import the mssql hook and operator into my dag but I keep getting this error from Airflow.
I'm currently importing with the newest syntax:
from airflow.providers.microsoft.mssql.hooks.mssql import MsSqlHook
from airflow.providers.microsoft.mssql.operators.mssql import MsSqlOperator
and I'm getting this import error:
File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
File "/opt/airflow/dags/hevo_dag.py", line 5, in <module>
from airflow.providers.microsoft.mssql.hooks.mssql import MsSqlHook
ModuleNotFoundError: No module named 'airflow.providers.microsoft.mssql'
To import the operator you need to do:
For Airflow>=2.0:
you need to use Mssql provider package:
pip install apache-airflow-providers-microsoft-mssql
For Airflow<2.0:
you need to use Mssql backport provider package:
pip install apache-airflow-backport-providers-microsoft-mssql
In your code it's the same import path (regardless of the package). Airflow backported the provider to ease migration from Airflow 1 to Airflow 2 so upon upgrading you will not need to change the import paths.
Exception has occurred: ModuleNotFoundError
No module named 'numba.decorators'
File "C:\Users\prasa\OneDrive\Desktop\Python\Untitled-1.py", line 3, in
import librosa
Install a previous version of numba since the newest versions deleted the decorators module.
you can use pip install numba==0.48
I am trying to call rpy2.robjects which is giving below mentioned error. I am using rpy2 package(3.4.5) in python 3.5.10. R installed version is 3.4.1.
import rpy2.robjects as ro
Traceback (most recent call last):
File "test.py", line 1, in <module>
import rpy2.robjects
File "/usr/local/lib64/python3.5/site-packages/rpy2/robjects/__init__.py", line 16, in <module>
import rpy2.rinterface as rinterface
File "/usr/local/lib64/python3.5/site-packages/rpy2/rinterface.py", line 13, in <module>
import rpy2.rinterface_lib._rinterface_capi as _rinterface
File "/usr/local/lib64/python3.5/site-packages/rpy2/rinterface_lib/_rinterface_capi.py", line 97
_cdata: FFI.CData
^
Syntax error : invalid syntax
[Rp2_error][1]
I am running this command from AWS EC2 instance where it is required to call an R function from python.
The same command is working in windows after setting R_HOME.
In EC2 instance, tried 2 paths for R_HOME,
os.environ['R_HOME'] = '/usr/lib64/R/'
or '/usr/bin/R'
The issue persists in both cases.
R.home is '/usr/lib64/R/' (R prompt)
rpy2 3.4.x requires Python >=3.6 as described in the documentation (installation section). The error you see is because Python 3.5 did not support typing.
You need to upgrade your Python version. Python 3.5 support ended 1 year ago and its not safe to use it - you may be exposed to unpatched security bugs. Given that support for 3.6 ends in two months I would recommend upgrading straight to 3.7 or 3.8.
I installed Apache airflow 1.9 from GitHub thanks to this command line on debian 9: pip install git+https://github.com/apache/incubator-airflow.git#v1-9-stable
However, I have an error during the airflow initdb caused by Fernet, do you know how to solve this issue?
INFO [alembic.runtime.migration] Running upgrade 947454bf1dff -> d2ae31099d61, Increase text size for MySQL (not relevant for other DBs' text types)
[2017-12-27 17:19:24,586] {models.py:643} ERROR - Failed to load fernet while encrypting value, using non-encrypted value.
Traceback (most recent call last):
File "/usr/local/lib/python2.7/dist-packages/airflow/models.py", line 639, in set_extra
fernet = get_fernet()
File "/usr/local/lib/python2.7/dist-packages/airflow/models.py", line 103, in get_fernet
raise AirflowException('Failed to import Fernet, it may not be installed')
AirflowException: Failed to import Fernet, it may not be installed
[2017-12-27 17:19:24,601] {models.py:643} ERROR - Failed to load fernet
And how can I specify extrapackage like in pip install apache-airflow[gcp-api] from my previous pip command install with GitHub?
How to install the latest 1.9.0RC too? I have an assertionError.
The answer marked as good have a broken link, if you have landed here as me and it continues broken, these steps have worked for me:
pip install cryptography
python -c "from cryptography.fernet import Fernet; print(Fernet.generate_key().decode())"
Add the generated key to the config file airflow.cfg, fernet_key = YOUR_GENERATED_KEY
During install from source you have to replace fernet_key in airflow.cfg such as you can find in the docs here.
In apache-airflow documentation, the script for generating fernet key is apparently wrong.
it says to use the following script.
from cryptography.fernet import Fernet
fernet_key= Fernet.generate_key()
print(fernet_key) # your fernet_key, keep it in secured place!
but it raises an exception at 'airflow initdb' command.
to solve this instead of Fernet.generate_key() use Fernet.generate_key().decode() as shown in #skozz answer.
I've just migrated from the Python 2.5 to the 2.7 runtime, and most of server I've written runs fine. However I'm occassionaly seeing this odd stack trace (I've hacked it down for brevity):
ERROR 2013-04-23 10:40:15,598 wsgi.py:235]
Traceback (most recent call last):
File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/lib/jinja2-2.6/jinja2/environment.py", line 894, in render
return self.environment.handle_exception(exc_info, True)
File "templates/querystart.html", line 30, in top-level template code
{% for session in sessions %}
ImportError: No module named _sqlite3
I've not imported sqlite3 myself, and I don't rely on it. The code that's calling it is pretty simple (perhaps bordering on the dumb side of things):
class UserIdQuery(BaseHandler):
def get(self):
sessionQuery=Session.all().order("userid")
template_values = {
'sessions': sessionQuery,
}
self.render_template('querystart.html',**template_values)
The above simply extends the below (taken from a very helpful migration tutorial)
class BaseHandler(webapp2.RequestHandler):
#webapp2.cached_property
def jinja2(self):
return jinja2.get_jinja2(app=self.app)
def render_template(self, filename, **template_args):
self.response.write(self.jinja2.render_template(filename, **template_args))
Anyone know what might be triggering the import error? Thanks,
Appengine uses sqlite for the datastore, my bet is your python2.7 installed on OSX is missing the sqlite binary library. Do a quick test, start a python interpreter (outside of appengine) and try to import sqlite3
$ python
Python 2.7.3 (default, Sep 26 2012, 21:53:58)
[GCC 4.7.2] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import sqlite3
>>>
If that fails you know you have an incomplete 2.7 runtime installed.
I was having the same problem and fixed it.
I was running python 2.7.3 that I installed from Ninite.com.
Uninstalled python, downloaded and installed 2.7.5 installer directly from python.org and problem has gone away.