Openstack Watcher complains about endpoint not found even though it is defined in the catalog - openstack

Question

I have the following scenario:
server: Ubuntu 20.04.3 LTS
Openstack: installed following the official guide
Watcher: 1:4.0.0-0ubuntu0.20.04.1 (installed also following the official wiki)
Everything works like a charm, however, when I run
root#controller:/etc/watcher# openstack optimize service list
Internal Server Error (HTTP 500)
root#controller:/etc/watcher#
and checked what is it about on watcher's log
2022-01-15 17:25:58.509 17960 INFO watcher-api [-] 10.0.0.11 "GET /v1/services HTTP/1.1" status: 500 len: 139 time: 0.0277412
2022-01-15 17:40:52.535 17960 INFO watcher-api [-] Traceback (most recent call last):
File "/usr/local/lib/python3.8/dist-packages/eventlet/wsgi.py", line 573, in handle_one_response
result = self.application(self.environ, start_response)
File "/usr/lib/python3/dist-packages/watcher/api/app.py", line 58, in __call__
return self.v1(environ, start_response)
File "/usr/lib/python3/dist-packages/watcher/api/middleware/auth_token.py", line 61, in __call__
return super(AuthTokenMiddleware, self).__call__(env, start_response)
File "/usr/local/lib/python3.8/dist-packages/webob/dec.py", line 129, in __call__
resp = self.call_func(req, *args, **kw)
File "/usr/local/lib/python3.8/dist-packages/webob/dec.py", line 193, in call_func
return self.func(req, *args, **kwargs)
File "/usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py", line 338, in __call__
response = self.process_request(req)
File "/usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py", line 659, in process_request
resp = super(AuthProtocol, self).process_request(request)
File "/usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py", line 409, in process_request
data, user_auth_ref = self._do_fetch_token(
File "/usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py", line 445, in _do_fetch_token
data = self.fetch_token(token, **kwargs)
File "/usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py", line 752, in fetch_token
data = self._identity_server.verify_token(
File "/usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/_identity.py", line 157, in verify_token
auth_ref = self._request_strategy.verify_token(
File "/usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/_identity.py", line 108, in _request_strategy
strategy_class = self._get_strategy_class()
File "/usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/_identity.py", line 130, in _get_strategy_class
if self._adapter.get_endpoint(version=klass.AUTH_VERSION):
File "/usr/local/lib/python3.8/dist-packages/keystoneauth1/adapter.py", line 291, in get_endpoint
return self.session.get_endpoint(auth or self.auth, **kwargs)
File "/usr/local/lib/python3.8/dist-packages/keystoneauth1/session.py", line 1233, in get_endpoint
return auth.get_endpoint(self, **kwargs)
File "/usr/local/lib/python3.8/dist-packages/keystoneauth1/identity/base.py", line 375, in get_endpoint
endpoint_data = self.get_endpoint_data(
File "/usr/local/lib/python3.8/dist-packages/keystoneauth1/identity/base.py", line 275, in get_endpoint_data
endpoint_data = service_catalog.endpoint_data_for(
File "/usr/local/lib/python3.8/dist-packages/keystoneauth1/access/service_catalog.py", line 462, in endpoint_data_for
raise exceptions.EndpointNotFound(msg)
keystoneauth1.exceptions.catalog.EndpointNotFound: internal endpoint for identity service in regionOne region not found
and the request on the webserver side
==> horizon_access.log <==
127.0.0.1 - - [15/Jan/2022:17:38:29 +0300] "GET /dashboard/project/api_access/view_credentials/ HTTP/1.1" 200 1027 "http://localhost/dashboard/project/api_access/" "Mozilla/5.0 (X11; Linux x86_64; rv:91.0) Gecko/20100101 Firefox/91.0"
10.0.0.11 - - [15/Jan/2022:17:38:30 +0300] "GET /identity/v3/auth/tokens HTTP/1.1" 200 5318 "-" "python-keystoneclient"
10.0.0.11 - - [15/Jan/2022:17:38:30 +0300] "GET /compute/v2.1/servers/detail?all_tenants=True&changes-since=2022-01-15T14%3A33%3A30.416004%2B00%3A00 HTTP/1.1" 200 433 "-" "python-novaclient"
10.0.0.11 - - [15/Jan/2022:17:40:52 +0300] "GET /identity HTTP/1.1" 300 569 "-" "openstacksdk/0.50.0 keystoneauth1/4.2.1 python-requests/2.23.0 CPython/3.8.10"
10.0.0.11 - - [15/Jan/2022:17:40:52 +0300] "POST /identity/v3/auth/tokens HTTP/1.1" 201 5316 "-" "openstacksdk/0.50.0 keystoneauth1/4.2.1 python-requests/2.23.0 CPython/3.8.10"
10.0.0.11 - - [15/Jan/2022:17:40:52 +0300] "POST /identity/v3/auth/tokens HTTP/1.1" 201 5320 "-" "watcher/unknown keystonemiddleware.auth_token/9.1.0 keystoneauth1/4.2.1 python-requests/2.23.0 CPython/3.8.10"
and on keystone side - I run it with some verbosity using the following command
/usr/bin/uwsgi --procname-prefix keystone --ini /etc/keystone/keystone-uwsgi-public.ini
I got the following log
DEBUG keystone.server.flask.request_processing.req_logging [None req-e422207d-b376-4e97-b20b-1d16144be4db None None] REQUEST_METHOD: `GET` {{(pid=20441) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}}
DEBUG keystone.server.flask.request_processing.req_logging [None req-e422207d-b376-4e97-b20b-1d16144be4db None None] SCRIPT_NAME: `/identity` {{(pid=20441) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}}
DEBUG keystone.server.flask.request_processing.req_logging [None req-e422207d-b376-4e97-b20b-1d16144be4db None None] PATH_INFO: `/` {{(pid=20441) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}}
[pid: 20441|app: 0|req: 1/1] 10.0.0.11 () {58 vars in 998 bytes} [Sat Jan 15 17:44:30 2022] GET /identity => generated 268 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 232 bytes (1 switches on core 0)
DEBUG keystone.server.flask.request_processing.req_logging [None req-cc547fb9-886e-4ed2-a3be-7e043004eed8 None None] REQUEST_METHOD: `POST` {{(pid=20440) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}}
DEBUG keystone.server.flask.request_processing.req_logging [None req-cc547fb9-886e-4ed2-a3be-7e043004eed8 None None] SCRIPT_NAME: `/identity` {{(pid=20440) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}}
DEBUG keystone.server.flask.request_processing.req_logging [None req-cc547fb9-886e-4ed2-a3be-7e043004eed8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=20440) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}}
DEBUG oslo_db.sqlalchemy.engines [None req-cc547fb9-886e-4ed2-a3be-7e043004eed8 None None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=20440) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:304}}
DEBUG passlib.handlers.bcrypt [None req-cc547fb9-886e-4ed2-a3be-7e043004eed8 None None] detected 'bcrypt' backend, version '3.2.0' {{(pid=20440) _load_backend_mixin /usr/local/lib/python3.8/dist-packages/passlib/handlers/bcrypt.py:567}}
DEBUG passlib.handlers.bcrypt [None req-cc547fb9-886e-4ed2-a3be-7e043004eed8 None None] 'bcrypt' backend lacks $2$ support, enabling workaround {{(pid=20440) _finalize_backend_mixin /usr/local/lib/python3.8/dist-packages/passlib/handlers/bcrypt.py:382}}
DEBUG keystone.auth.core [None req-cc547fb9-886e-4ed2-a3be-7e043004eed8 None None] MFA Rules not processed for user `97eec1465cdc4e41b5c0ba48a1b39cc2`. Rule list: `[]` (Enabled: `True`). {{(pid=20440) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}}
DEBUG keystone.common.fernet_utils [None req-cc547fb9-886e-4ed2-a3be-7e043004eed8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=20440) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}}
[pid: 20440|app: 0|req: 1/2] 10.0.0.11 () {62 vars in 1095 bytes} [Sat Jan 15 17:44:30 2022] POST /identity/v3/auth/tokens => generated 4862 bytes in 125 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0)
DEBUG keystone.server.flask.request_processing.req_logging [None req-0584fbcc-66c5-4fba-9d8a-ea8ad2d40c5d None None] REQUEST_METHOD: `GET` {{(pid=20441) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}}
DEBUG keystone.server.flask.request_processing.req_logging [None req-0584fbcc-66c5-4fba-9d8a-ea8ad2d40c5d None None] SCRIPT_NAME: `/identity` {{(pid=20441) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}}
DEBUG keystone.server.flask.request_processing.req_logging [None req-0584fbcc-66c5-4fba-9d8a-ea8ad2d40c5d None None] PATH_INFO: `/` {{(pid=20441) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}}
[pid: 20441|app: 0|req: 2/3] 10.0.0.11 () {58 vars in 1033 bytes} [Sat Jan 15 17:44:30 2022] GET /identity => generated 268 bytes in 2 msecs (HTTP/1.1 300) 6 headers in 232 bytes (1 switches on core 0)
DEBUG keystone.server.flask.request_processing.req_logging [None req-f096d017-66d0-4baa-8414-2596d0869005 None None] REQUEST_METHOD: `POST` {{(pid=20440) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}}
DEBUG keystone.server.flask.request_processing.req_logging [None req-f096d017-66d0-4baa-8414-2596d0869005 None None] SCRIPT_NAME: `/identity` {{(pid=20440) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}}
DEBUG keystone.server.flask.request_processing.req_logging [None req-f096d017-66d0-4baa-8414-2596d0869005 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=20440) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}}
DEBUG keystone.auth.core [None req-f096d017-66d0-4baa-8414-2596d0869005 None None] MFA Rules not processed for user `c5c42a1a942e48fd9b735ea9c6a11ed0`. Rule list: `[]` (Enabled: `True`). {{(pid=20440) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}}
DEBUG keystone.common.fernet_utils [None req-f096d017-66d0-4baa-8414-2596d0869005 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=20440) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}}
[pid: 20440|app: 0|req: 2/4] 10.0.0.11 () {62 vars in 1130 bytes} [Sat Jan 15 17:44:30 2022] POST /identity/v3/auth/tokens => generated 4866 bytes in 26 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0)
So the first thing I did is to check the catalog
openstack catalog list
----
| keystone | identity | RegionOne |
| | | internal: http://controller/identity |
| | | RegionOne |
| | | public: http://controller/identity |
| | | RegionOne |
| | | admin: http://controller/identity |
| | | |
---
My question is: do I need to create a specific (another) internal endpoint for the identity service and where should I declare it for the watcher-api to find it?
EDIT: Following #Larsks comment, I changed the credentials used on watcher.conf by username=admin (the admin user) and the corresponding password. Openstack optimize service list gave back the following
WARNING keystonemiddleware.auth_token [-] Identity response: {"error":{"code":401,"message":"The request you have made requires authentication.","title":"Unauthorized"}}
: keystoneauth1.exceptions.http.Unauthorized: The request you have made requires authentication. (HTTP 401) (Request-ID: req-56b63a60-1ba2-4f12-93c0-e7c7d1a1769c)
2022-01-15 19:04:17.424 28742 CRITICAL keystonemiddleware.auth_token [-] Unable to validate token: Identity server rejected authorization necessary to fetch token data: keystonemiddleware.auth_token._exceptions.ServiceError: Identity server rejected authorization necessary to fetch token data