I have installed OpenStack with devstack scripts in a two-node configuration, i.e. a controller/network node and a separate compute node. Everything seems have started properly, I see the services running on the nodes (if connect to screen sessions). However when trying to start an instance via dashboard, it fails with the following in the logs:
ERROR oslo.messaging._drivers.common [req-892e7b17-49dc-4ce5-a193-ceaf547c97eb admin demo]
Traceback (most recent call last):
File "/usr/lib/python2.7/site-packages/oslo/messaging/rpc/dispatcher.py", line 137, in _dispatch_and_reply incoming.message))
File "/usr/lib/python2.7/site-packages/oslo/messaging/rpc/dispatcher.py", line 180, in _dispatch
return self._do_dispatch(endpoint, method, ctxt, args)
File "/usr/lib/python2.7/site-packages/oslo/messaging/rpc/dispatcher.py", line 126, in _do_dispatch
result = getattr(endpoint, method)(ctxt, **new_args)
File "/usr/lib/python2.7/site-packages/oslo/messaging/rpc/server.py", line 139, in inner
return func(*args, **kwargs)
File "/opt/stack/nova/nova/scheduler/manager.py", line 175, in select_destinations
filter_properties)
File "/opt/stack/nova/nova/scheduler/filter_scheduler.py", line 147, in select_destinations
filter_properties)
File "/opt/stack/nova/nova/scheduler/filter_scheduler.py", line 276, in _schedule
filter_properties, index=num)
File "/opt/stack/nova/nova/scheduler/host_manager.py", line 359, in get_filtered_hosts
filter_classes = self._choose_host_filters(filter_class_names)
File "/opt/stack/nova/nova/scheduler/host_manager.py", line 309, in _choose_host_filters
raise exception.SchedulerHostFilterNotFound(filter_name=msg)
SchedulerHostFilterNotFound: Scheduler Host Filter could not be found.
If somebody faced this issue before, could you explain how to fix it?
Related
I've recently created custom timetable. Worked perfectly locally (python==3.9.12, airflow==2.3.0), so decided to upload it to plugins folder in my Cloud Composer (version==1.18.11, airflow==2.2.5). While scheduler picks up timetable and dag is run based on it, trying to open dag in UI throws me this error window:
Something bad has happened.
Airflow is used by many users, and it is very likely that others had similar problems and you can easily find
a solution to your problem.
Consider following these steps:
* gather the relevant information (detailed logs with errors, reproduction steps, details of your deployment)
* find similar issues using:
* GitHub Discussions
* GitHub Issues
* Stack Overflow
* the usual search engine you use on a daily basis
* if you run Airflow on a Managed Service, consider opening an issue using the service support channels
* if you tried and have difficulty with diagnosing and fixing the problem yourself, consider creating a bug report.
Make sure however, to include all relevant details and results of your investigation so far.
Python version: 3.8.12
Airflow version: 2.2.5+composer
Node: 67b211ed8faa
-------------------------------------------------------------------------------
Traceback (most recent call last):
File "/opt/python3.8/lib/python3.8/site-packages/flask/app.py", line 2447, in wsgi_app
response = self.full_dispatch_request()
File "/opt/python3.8/lib/python3.8/site-packages/flask/app.py", line 1952, in full_dispatch_request
rv = self.handle_user_exception(e)
File "/opt/python3.8/lib/python3.8/site-packages/flask/app.py", line 1821, in handle_user_exception
reraise(exc_type, exc_value, tb)
File "/opt/python3.8/lib/python3.8/site-packages/flask/_compat.py", line 39, in reraise
raise value
File "/opt/python3.8/lib/python3.8/site-packages/flask/app.py", line 1950, in full_dispatch_request
rv = self.dispatch_request()
File "/opt/python3.8/lib/python3.8/site-packages/flask/app.py", line 1936, in dispatch_request
return self.view_functions[rule.endpoint](**req.view_args)
File "/opt/python3.8/lib/python3.8/site-packages/airflow/www/auth.py", line 51, in decorated
return func(*args, **kwargs)
File "/opt/python3.8/lib/python3.8/site-packages/airflow/www/decorators.py", line 108, in view_func
return f(*args, **kwargs)
File "/opt/python3.8/lib/python3.8/site-packages/airflow/www/decorators.py", line 71, in wrapper
return f(*args, **kwargs)
File "/opt/python3.8/lib/python3.8/site-packages/airflow/utils/session.py", line 70, in wrapper
return func(*args, session=session, **kwargs)
File "/opt/python3.8/lib/python3.8/site-packages/airflow/www/views.py", line 2328, in tree
dag = current_app.dag_bag.get_dag(dag_id)
File "/opt/python3.8/lib/python3.8/site-packages/airflow/utils/session.py", line 70, in wrapper
return func(*args, session=session, **kwargs)
File "/opt/python3.8/lib/python3.8/site-packages/airflow/models/dagbag.py", line 186, in get_dag
self._add_dag_from_db(dag_id=dag_id, session=session)
File "/opt/python3.8/lib/python3.8/site-packages/airflow/models/dagbag.py", line 261, in _add_dag_from_db
dag = row.dag
File "/opt/python3.8/lib/python3.8/site-packages/airflow/models/serialized_dag.py", line 180, in dag
dag = SerializedDAG.from_dict(self.data) # type: Any
File "/opt/python3.8/lib/python3.8/site-packages/airflow/serialization/serialized_objects.py", line 951, in from_dict
return cls.deserialize_dag(serialized_obj['dag'])
File "/opt/python3.8/lib/python3.8/site-packages/airflow/serialization/serialized_objects.py", line 877, in deserialize_dag
v = _decode_timetable(v)
File "/opt/python3.8/lib/python3.8/site-packages/airflow/serialization/serialized_objects.py", line 167, in _decode_timetable
raise _TimetableNotRegistered(importable_string)
airflow.serialization.serialized_objects._TimetableNotRegistered: Timetable class '<enter_your_timetable_plugin_name>.<enter_your_timetable_class_name>' is not registered
Going to window Plugins shows that no plugins are added (both Cloud Composer==2.0.15, airflow==2.2.5) and my local setup uploads plugin properly.
What's really interesting that while having same airflow version, both versions of Cloud Composer works differently.
I don't override any of default airflow variables, nor that should impact anything that's described here.
Many many thanks for any suggestions.
So it looks like my install of apache airflow on a Google Compute Engine instance broke down. Everything was working great and then two days ago all the DAG runs show up stuck in a running state. I am using LocalExecutioner.
When I try to look at the log I get this error:
* Log file isn't local.
* Fetching here: http://:8793/log/collector/aa_main_combined_collector/2017-12-15T09:00:00
*** Failed to fetch log file from worker.
I didn't touch a setting anywhere. I looked through all the config files and I scanned the logs and I see this error
[2017-12-16 20:08:42,558] {jobs.py:355} DagFileProcessor0 ERROR - Got an exception! Propagating...
Traceback (most recent call last):
File "/usr/local/lib/python3.4/dist-packages/airflow/jobs.py", line 347, in helper
pickle_dags)
File "/usr/local/lib/python3.4/dist-packages/airflow/utils/db.py", line 53, in wrapper
result = func(*args, **kwargs)
File "/usr/local/lib/python3.4/dist-packages/airflow/jobs.py", line 1584, in process_file
self._process_dags(dagbag, dags, ti_keys_to_schedule)
File "/usr/local/lib/python3.4/dist-packages/airflow/jobs.py", line 1173, in _process_dags
dag_run = self.create_dag_run(dag)
File "/usr/local/lib/python3.4/dist-packages/airflow/utils/db.py", line 53, in wrapper
result = func(*args, **kwargs)
File "/usr/local/lib/python3.4/dist-packages/airflow/jobs.py", line 763, in create_dag_run
last_scheduled_run = qry.scalar()
File "/usr/local/lib/python3.4/dist-packages/sqlalchemy/orm/query.py", line 2843, in scalar
ret = self.one()
File "/usr/local/lib/python3.4/dist-packages/sqlalchemy/orm/query.py", line 2814, in one
ret = self.one_or_none()
File "/usr/local/lib/python3.4/dist-packages/sqlalchemy/orm/query.py", line 2784, in one_or_none
ret = list(self)
File "/usr/local/lib/python3.4/dist-packages/sqlalchemy/orm/query.py", line 2855, in iter
return self._execute_and_instances(context)
File "/usr/local/lib/python3.4/dist-packages/sqlalchemy/orm/query.py", line 2878, in _execute_and_instances
result = conn.execute(querycontext.statement, self._params)
File "/usr/local/lib/python3.4/dist-packages/sqlalchemy/engine/base.py", line 945, in execute
return meth(self, multiparams, params)
File "/usr/local/lib/python3.4/dist-packages/sqlalchemy/sql/elements.py", line 263, in _execute_on_connection
return connection._execute_clauseelement(self, multiparams, params)
File "/usr/local/lib/python3.4/dist-packages/sqlalchemy/engine/base.py", line 1053, in _execute_clauseelement
compiled_sql, distilled_params
File "/usr/local/lib/python3.4/dist-packages/sqlalchemy/engine/base.py", line 1189, in _execute_context
context)
File "/usr/local/lib/python3.4/dist-packages/sqlalchemy/engine/base.py", line 1405, in _handle_dbapi_exception
util.reraise(*exc_info)
File "/usr/local/lib/python3.4/dist-packages/sqlalchemy/util/compat.py", line 187, in reraise
raise value
File "/usr/local/lib/python3.4/dist-packages/sqlalchemy/engine/base.py", line 1182, in _execute_context
context)
File "/usr/local/lib/python3.4/dist-packages/sqlalchemy/engine/default.py", line 470, in do_execute
cursor.execute(statement, parameters)
File "/usr/local/lib/python3.4/dist-packages/airflow/bin/cli.py", line 69, in sigint_handler
sys.exit(0)
SystemExit: 0
Any thoughts out there?
I solved this problem though in doing so I discovered another problem.
Long and short of it as soon as I manually started the scheduler, everything worked again. It appears the problem was that the scheduler did not get restarted correctly after a system reboot.
I have scheduler running through SystemD. The Webserver .service works fine. However I do notice that the scheduler .service continually restarts. It appears there is an issue there I need to resolve. This part of it is solved for now.
Look at the log URL, verify if it ends with a date with special characters +:
&execution_date=2018-02-23T08:00:00+00:00
This was fixed here.
You can replace the + for -, or replace all special characters, in my case:
&execution_date=2018-02-23T08%3A00%3A00%2B00%3A00
This happens here.
The FileTaskHandler can not load the log from local disk, and try to load from worker.
Another thing that could be causing this error is the exclusion of the airflow/logs folder or the sub folders inside it.
I've setup a database connection using sql_alchemy_conn = ibm_db_sa://{USERNAME}:{PASSWORD}#{HOST}:50000/airflow in the airflow.cfg file.
When I run airflow initdb, it pops up KeyError: 'ibm_db_sa'. How can I use a DB2 connection with Airflow?
===============
Here is more specific error message:
airflow initdb
[2017-02-01 15:55:57,135] {__init__.py:36} INFO - Using executor SequentialExecutor
DB: ibm_db_sa://db2inst1:***#localhost:50000/airflow
[2017-02-01 15:55:58,151] {db.py:222} INFO - Creating tables
Traceback (most recent call last):
File "/opt/anaconda/bin/airflow", line 15, in <module>
args.func(args)
File "/opt/anaconda/lib/python2.7/site-packages/airflow/bin/cli.py", line 524, in initdb
db_utils.initdb()
File "/opt/anaconda/lib/python2.7/site-packages/airflow/utils/db.py", line 106, in initdb
upgradedb()
File "/opt/anaconda/lib/python2.7/site-packages/airflow/utils/db.py", line 230, in upgradedb
command.upgrade(config, 'heads')
File "/opt/anaconda/lib/python2.7/site-packages/alembic/command.py", line 174, in upgrade
script.run_env()
File "/opt/anaconda/lib/python2.7/site-packages/alembic/script/base.py", line 416, in run_env
util.load_python_file(self.dir, 'env.py')
File "/opt/anaconda/lib/python2.7/site-packages/alembic/util/pyfiles.py", line 93, in load_python_file
module = load_module_py(module_id, path)
File "/opt/anaconda/lib/python2.7/site-packages/alembic/util/compat.py", line 79, in load_module_py
mod = imp.load_source(module_id, path, fp)
File "/opt/anaconda/lib/python2.7/site-packages/airflow/migrations/env.py", line 74, in <module>
run_migrations_online()
File "/opt/anaconda/lib/python2.7/site-packages/airflow/migrations/env.py", line 65, in run_migrations_online
compare_type=COMPARE_TYPE,
File "<string>", line 8, in configure
File "/opt/anaconda/lib/python2.7/site-packages/alembic/runtime/environment.py", line 773, in configure
opts=opts
File "/opt/anaconda/lib/python2.7/site-packages/alembic/runtime/migration.py", line 159, in configure
return MigrationContext(dialect, connection, opts, environment_context)
File "/opt/anaconda/lib/python2.7/site-packages/alembic/runtime/migration.py", line 103, in __init__
self.impl = ddl.DefaultImpl.get_by_dialect(dialect)(
File "/opt/anaconda/lib/python2.7/site-packages/alembic/ddl/impl.py", line 65, in get_by_dialect
return _impls[dialect.name]
KeyError: 'ibm_db_sa'
Did you install the required package for db2? Eg. pip install ibm_db_sa. By the way the pypi page lists that it is only compatible with python 3.
Please note that db2 is not officially supported as a backend for Airflow.
I am running a pretty much unmodified instance of Plone 5.0.
I created a lot of folders today, one of which was named "Layout" in a parent folder "Design", which was in the root folder.
This Layout folder caused an error in the view of the Design folder. All I see is the error page
We’re sorry, but there seems to be an error…
The error has been logged as entry number 1470387402.080.1605824509.
If you need to report this to the
Site Administration, please include this entry number in your message.
The error log of Plone has the following entry:
Exception Type
RuntimeError
Exception Value
maximum recursion depth exceeded
Traceback (innermost last):
Module ZPublisher.Publish, line 127, in publish
Module ZPublisher.BaseRequest, line 444, in traverse
Module Products.CMFCore.DynamicType, line 147, in __before_publishing_traverse__
Module Products.CMFDynamicViewFTI.fti, line 236, in queryMethodID
Module Products.CMFDynamicViewFTI.fti, line 197, in defaultView
Module Products.CMFPlone.PloneTool, line 771, in browserDefault
Module Products.CMFDynamicViewFTI.browserdefault, line 99, in getLayout
Module Products.CMFDynamicViewFTI.fti, line 136, in getViewMethod
Module Products.CMFDynamicViewFTI.browserdefault, line 72, in __call__
Module Products.CMFDynamicViewFTI.browserdefault, line 72, in __call__
Module Products.CMFDynamicViewFTI.browserdefault, line 72, in __call__
Module Products.CMFDynamicViewFTI.browserdefault, line 72, in __call__
Module Products.CMFDynamicViewFTI.browserdefault, line 72, in __call__
and goes on for a couple of dozen identical calls.
This only happens with this specific folder. I deleted it and created it again: same error. All other folders work fine, even much deeper folder structures work fine.
Any idea what is wrong with this specific folder name?
Sometimes this happen with Plone due to acquisition.
Some names are reserved (looking at your traceback the issue seems related to the view selection, and Layout is in fact a dangerous name).
Plone itself protects you from creating some bad ids but it can cover all the cases.
Another common case with this type of issue is calling a catalog index "data", or calling a content like a catalog index.
Just use a different name.
After 3.3 -> 4.1 migration I get exception on the resulting page
File "/fast/buildout-cache/eggs/plone.app.viewletmanager-2.0.2-py2.6.egg/plone/app/viewletmanager/manager.py", line 85, in render
return u'\n'.join([viewlet.render() for viewlet in self.viewlets])
File "/fast/buildout-cache/eggs/plone.memoize-1.1.1-py2.6.egg/plone/memoize/volatile.py", line 281, in replacement
cached_value = cache.get(key, _marker)
File "/fast/buildout-cache/eggs/plone.memoize-1.1.1-py2.6.egg/plone/memoize/ram.py", line 124, in get
return self.__getitem__(key)
File "/fast/buildout-cache/eggs/plone.memoize-1.1.1-py2.6.egg/plone/memoize/ram.py", line 166, in __getitem__
MARKER)
File "/fast/buildout-cache/eggs/zope.ramcache-1.0-py2.6.egg/zope/ramcache/ram.py", line 107, in query
s = self._getStorage()
File "/fast/buildout-cache/eggs/zope.ramcache-1.0-py2.6.egg/zope/ramcache/ram.py", line 122, in _getStorage
cacheId = self._cacheId
AttributeError: 'RAMCache' object has no attribute '_cacheId'
Looks like RAMCache object is in invalid state.
Also before this seeing in logs:
2012-06-21 16:42:54 INFO plone.app.upgrade Ran upgrade step: Miscellaneous
2012-06-21 16:42:54 INFO plone.app.upgrade End of upgrade path, migration has finished
2012-06-21 16:42:54 INFO plone.app.upgrade Your Plone instance is now up-to-date.
2012-06-21 16:43:02 ERROR txn.4553572352 Error in tpc_abort() on manager <Connection at 10be48490>
Traceback (most recent call last):
File "/fast/buildout-cache/eggs/transaction-1.1.1-py2.6.egg/transaction/_transaction.py", line 484, in _cleanup
rm.tpc_abort(self)
File "/fast/buildout-cache/eggs/ZODB3-3.10.5-py2.6-macosx-10.7-x86_64.egg/ZODB/Connection.py", line 730, in tpc_abort
self._storage.tpc_abort(transaction)
File "/fast/buildout-cache/eggs/ZODB3-3.10.5-py2.6-macosx-10.7-x86_64.egg/ZEO/ClientStorage.py", line 1157, in tpc_abort
self._server.tpc_abort(id(txn))
File "/fast/buildout-cache/eggs/ZODB3-3.10.5-py2.6-macosx-10.7-x86_64.egg/ZEO/ServerStub.py", line 255, in tpc_abort
self.rpc.call('tpc_abort', id)
File "/fast/buildout-cache/eggs/ZODB3-3.10.5-py2.6-macosx-10.7-x86_64.egg/ZEO/zrpc/connection.py", line 768, in call
raise inst # error raised by server
OSError: [Errno 2] No such file or directory: '/fast/xxx-2012/var/blobstorage/0x00/0x00/0x00/0x00/0x00/0x07/0xdc/0x55/0x00FWigqp.tmp-'
2012-06-21 16:43:03 ERROR Zope.SiteErrorLog 1340286183.10.000607291180815 http://localhost:9666/xxx/##plone-upgrade
Traceback (innermost last):
Module ZPublisher.Publish, line 134, in publish
Module Zope2.App.startup, line 301, in commit
Module transaction._manager, line 89, in commit
Module transaction._transaction, line 329, in commit
Module transaction._transaction, line 446, in _commitResources
Module ZODB.Connection, line 781, in tpc_vote
Module ZEO.ClientStorage, line 1098, in tpc_vote
Module ZEO.ClientStorage, line 929, in _check_serials
IOError: [Errno 2] No such file or directory: '/fast/xxx-2012/var/blobstorage/0x00/0x00/0x00/0x00/0x00/0x07/0xdd/0xca/0x009kWNYQ.tmp-'
Why this would might happen?
Any pointers how to reinitialize RAMCache object?
RAMCache is first time referred by FaviconViewlet which is using #memoize deorator and it leads to this error.
Well, your migration obviously did not complete successfully, based on the traceback. So I would focus on figuring out why it failed, rather than working around things like the broken RAMCache which are likely a result of the migration not having run.
The traceback indicates that it broke while trying to abort the transaction...so you'll probably need to do some debugging to determine what caused it to try to abort, since that's not indicated in the logs you pasted.