Given below json:
{ "Model" : "level1" }
what is the right combination of message_filtering_match_values and message_filtering_config values? I try below but it fails:
model_operator = SQSSensor(
task_id='model_operator',
dag=dag,
sqs_queue='https://sqs.somewhere/somequeue.fifo',
aws_conn_id='aws_default',
message_filtering='jsonpath',
message_filtering_config='Model[*]',
message_filtering_match_values=['level1'],
mode='reschedule')
Error message is:
Broken DAG: [/usr/local/airflow/dags/test_dag.py] Traceback (most recent call last):
File "/usr/local/lib/python3.7/site-packages/airflow/utils/decorators.py", line 94, in wrapper
result = func(*args, **kwargs)
File "/usr/local/lib/python3.7/site-packages/airflow/models/baseoperator.py", line 414, in __init__
"arguments were:\n**kwargs: {k}".format(c=self.__class__.__name__, k=kwargs, t=task_id),
airflow.exceptions.AirflowException: Invalid arguments were passed to SQSSensor (task_id: model_operator). Invalid arguments were:
**kwargs: {'message_filtering': 'jsonpath', 'message_filtering_config': 'Model[*]', 'message_filtering_match_values': ['level1']}
The message_filtering / message_filtering_config / message_filtering_match_values were added recently in PR it was released in Amazon provider version 2.2.0
From the traceback we can see that these parameters are not recognized by the operator which means that you are running an older version of the Amazon provider.
You should upgrade the Amazon provider to the latest version.
pip install apache-airflow-providers-amazon --upgrade
It's also recommended to read the documentation about constraint files.
You didn't mention what Airflow version you are running nor what version of the Amazon provider so note to read the change logs in case you are upgrading major version.
Related
I try to connect to Azure Data Lake using Airflow. I use Airflow connection via the Web UI.
When I try to connect using the test button, I get an error Bad Request. As seen below
I use the correct UUIDs. These UUIDs have been verified in other cases. I also checked the firewall.
When I execute the DAG, I use the Azure Data Lake connection id to check if a file exists: If I apply the method as described here: What is the best way to check if a file exists on an Azure Datalake using Apache Airflow?
This is the error I get
[2022-05-06, 17:27:33 UTC] {log.py:127} ERROR - 99ec1d77-e91c-4fd3-a1c7-fa751ca1e779 - OAuth2Client:The token response from the server is unparseable as JSON: ***
Traceback (most recent call last):
File "/opt/airflow/lib/python3.8/site-packages/adal/oauth2_client.py", line 168, in _validate_token_response
wire_response = json.loads(body)
File "/usr/lib/python3.8/json/init.py", line 357, in loads
return _default_decoder.decode(s)
File "/usr/lib/python3.8/json/decoder.py", line 337, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
File "/usr/lib/python3.8/json/decoder.py", line 355, in raw_decode
raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 3 column 1 (char 4)
[2022-05-06, 17:27:33 UTC] {log.py:127} ERROR - 99ec1d77-e91c-4fd3-a1c7-fa751ca1e779 - OAuth2Client:Error validating get token response: ***
Traceback (most recent call last):
File "/opt/airflow/lib/python3.8/site-packages/adal/oauth2_client.py", line 238, in _handle_get_token_response
return self._validate_token_response(body)
File "/opt/airflow/lib/python3.8/site-packages/adal/oauth2_client.py", line 168, in _validate_token_response
Authenticating to Azure Data Lake is by token credentials i.e. add specific credentials (client_id, secret, tenant) and account name to the Airflow connection.
Information about how to set it up can be found in this doc.
You can see code example in the source code test function.
Other method of authentication are currently not supported.
I was trying to get the connection running using the Airflow implementation. My impression was that it was buggy and did not work out well. The above situation happened with Airflow 2.2.5. When I upgraded to Airflow 2.3.0, the test button was grayed out.
The final solution was to use Access Tokens instead.
I created my own docs for Read the Docs. See my repository
Some of my docs files are jupyter notebook so I used nbshpinx for it.
In my computer I installed all the dependencies and it works great when I use make html.
However, Read the docs throws the error:
Running Sphinx v1.8.5
loading translations [en]... done
Traceback (most recent call last):
File "/home/docs/checkouts/readthedocs.org/user_builds/complex-valued-neural-networks/envs/latest/lib/python3.7/site-packages/sphinx/registry.py", line 472, in load_extension
mod = __import__(extname, None, None, ['setup'])
ModuleNotFoundError: No module named 'nbsphinx'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/docs/checkouts/readthedocs.org/user_builds/complex-valued-neural-networks/envs/latest/lib/python3.7/site-packages/sphinx/cmd/build.py", line 303, in build_main
args.tags, args.verbosity, args.jobs, args.keep_going)
File "/home/docs/checkouts/readthedocs.org/user_builds/complex-valued-neural-networks/envs/latest/lib/python3.7/site-packages/sphinx/application.py", line 228, in __init__
self.setup_extension(extension)
File "/home/docs/checkouts/readthedocs.org/user_builds/complex-valued-neural-networks/envs/latest/lib/python3.7/site-packages/sphinx/application.py", line 449, in setup_extension
self.registry.load_extension(self, extname)
File "/home/docs/checkouts/readthedocs.org/user_builds/complex-valued-neural-networks/envs/latest/lib/python3.7/site-packages/sphinx/registry.py", line 475, in load_extension
raise ExtensionError(__('Could not import extension %s') % extname, err)
sphinx.errors.ExtensionError: Could not import extension nbsphinx (exception: No module named 'nbsphinx')
Extension error:
Could not import extension nbsphinx (exception: No module named 'nbsphinx')
Following this tutorial I created two yml files and the error changed to:
Error
Problem in your project's configuration. Invalid "conda.environment": environment not found
Solved it!
I followed this tutorial
I added in readthedocs.yml:
python:
version: 3
install:
- requirements: docs/requirements.txt
system_packages: true
And then in docs/requirements.txt:
ipykernel
nbsphinx
If questions you can always check the repository where I do it.
I've completed a manual installation of Eucalpytus 4.4.4 but when I try to use a web browser to reach the eucaconsole (running on the same host as CLC/UFS) I get a 502 Bad Gateway Error.
I'm focusing on this error in the eucaconsole.log What does it mean and how can I update WebOb?
pkg_resources.VersionConflict: (WebOb 1.2.3 (/usr/lib/python2.7/site-packages), Requirement.parse('WebOb>=1.3.1'))
Eucaconsole_startup.log:
Traceback (most recent call last):
File "/bin/eucaconsole", line 106, in <module>
daemonize(start_console)
File "/bin/eucaconsole", line 61, in daemonize
func()
File "/bin/eucaconsole", line 73, in start_console
load_entry_point('pyramid', 'console_scripts', 'pserve')(args)
File "/usr/lib/python2.7/site-packages/pkg_resources.py", line 378, in load_entry_point
return get_distribution(dist).load_entry_point(group, name)
File "/usr/lib/python2.7/site-packages/pkg_resources.py", line 2566, in load_entry_point
return ep.load()
File "/usr/lib/python2.7/site-packages/pkg_resources.py", line 2259, in load
if require: self.require(env, installer)
File "/usr/lib/python2.7/site-packages/pkg_resources.py", line 2272, in require
working_set.resolve(self.dist.requires(self.extras),env,installer)))
File "/usr/lib/python2.7/site-packages/pkg_resources.py", line 630, in resolve
raise VersionConflict(dist,req) # XXX put more info here
pkg_resources.VersionConflict: (WebOb 1.2.3 (/usr/lib/python2.7/site-packages), Requirement.parse('WebOb>=1.3.1'))
I've edited this post to remove info and focus on the clear WebOb version error.
This issue was solved below. Adding a note that the downlevel python-webob version is actually a requirement of the midonet client installation. So if you expect to run VPCMIDO and have the midonet gateway on your CLC you'll have to run your eucaconsole elsewhere.
This was resolved by removing an older python-webob package to ensure that the newer python-webob1.4 package from epel was used.
Related issue in github:
https://github.com/Corymbia/eucalyptus/issues/124
Selinux problem. Run the following in the your CLC/UFS machine
setsebool -P httpd_can_network_connect 1
Its better to flush your iptables during installation.
when I try to install nodecellar with Cloudify,I am getting the following error
2015-07-13T17:31:03 LOG <nodecellar> [mongod_a50aa.configure] ERROR: Exception raised on operation [script_runner.tasks.run] invocation
Traceback (most recent call last):
File "/root/cloudify.host_dba5c/env/local/lib/python2.7/site-packages/cloudify/decorators.py", line 125, in wrapper
result = func(*args, **kwargs)
File "/root/cloudify.host_dba5c/env/local/lib/python2.7/site-packages/script_runner/tasks.py", line 58, in run
return process_execution(script_func, script_path, ctx, process)
File "/root/cloudify.host_dba5c/env/local/lib/python2.7/site-packages/script_runner/tasks.py", line 74, in process_execution
script_func(script_path, ctx, process)
File "/root/cloudify.host_dba5c/env/local/lib/python2.7/site-packages/script_runner/tasks.py", line 143, in execute
stderr_consumer.buffer.getvalue())
How can I fix this problem?
This exception is raised by the Cloudify Script Plugin you ran a script, which exited with a non-zero error code. Here is the source of that error.
The script that returned non-zero code is that script which is mapped to the configure operation on the mongod node. Which script that is depends on the version of the Nodecellar blueprint that you are using.
I can't give a more detailed answer without information regarding the specific blueprint version, which Cloudify version you have installed, details about your provider (local, Vagrant, Openstack, AWS), and OS (Ubuntu, Centos, etc).
What is current best practice for profiling Plone/Zope applications on a Plone 4.1/Zope 2.13 system?
Products.ZopeProfiler and the Plone.org version of CallProfiler both fail at install time with:
Traceback (most recent call last):
File "/home/derek/plone4/buildout-cache/eggs/Zope2-2.13.10-py2.6.egg/OFS/Application.py", line 689, in install_product
initmethod(context)
File "/home/derek/plone4/buildout-cache/eggs/Products.ZopeProfiler-2.0.1-py2.6.egg/Products/ZopeProfiler/__init__.py", line 8, in initialize
control_panel = context._ProductContext__app.Control_Panel
AttributeError: 'NoneType' object has no attribute 'Control_Panel'
z3c.profiler wants to install a lot of apparently-incompatible dependencies (not surprising since it's intended for Zope3).
collective.profiler looks promising, and installs, but doesn't let me profile the __call__ method of a View, because the method doesn't have a __dict__ attribute.
Have a look at repoze.profile and it's documentation. It runs as WSGI middleware component which aggregates profiling data across all requests to the WSGI application.
The ZopeProfiler in Version 2.0.2 works just fine with Plone 4.1