Related
We deployed Apache Airflow 2.3.3 to Azure.
Webserver - Web App
Scheduler - ACI
Celery Worker - ACI
We were seeing errors on the Celery ACI console related to Postgres and Redis connection timeouts as below
[2022-09-22 18:55:50,650: WARNING/ForkPoolWorker-15] Failed operation _store_result. Retrying 2 more times.
Traceback (most recent call last):
File "/home/airflow/.local/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 1803, in _execute_context
cursor, statement, parameters, context
File "/home/airflow/.local/lib/python3.7/site-packages/sqlalchemy/engine/default.py", line 719, in do_execute
cursor.execute(statement, parameters)
psycopg2.DatabaseError: could not receive data from server: Connection timed out
SSL SYSCALL error: Connection timed out
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/home/airflow/.local/lib/python3.7/site-packages/celery/backends/database/__init__.py", line 47, in _inner
return fun(*args, **kwargs)
File "/home/airflow/.local/lib/python3.7/site-packages/celery/backends/database/__init__.py", line 117, in _store_result
task = list(session.query(self.task_cls).filter(self.task_cls.task_id == task_id))
File "/home/airflow/.local/lib/python3.7/site-packages/sqlalchemy/orm/query.py", line 2887, in __iter__
return self._iter().__iter__()
File "/home/airflow/.local/lib/python3.7/site-packages/sqlalchemy/orm/query.py", line 2897, in _iter
execution_options={"_sa_orm_load_options": self.load_options},
File "/home/airflow/.local/lib/python3.7/site-packages/sqlalchemy/orm/session.py", line 1689, in execute
result = conn._execute_20(statement, params or {}, execution_options)
File "/home/airflow/.local/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 1614, in _execute_20
return meth(self, args_10style, kwargs_10style, execution_options)
File "/home/airflow/.local/lib/python3.7/site-packages/sqlalchemy/sql/elements.py", line 326, in _execute_on_connection
self, multiparams, params, execution_options
File "/home/airflow/.local/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 1491, in _execute_clauseelement
cache_hit=cache_hit,
File "/home/airflow/.local/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 1846, in _execute_context
e, statement, parameters, cursor, context
File "/home/airflow/.local/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 2027, in _handle_dbapi_exception
sqlalchemy_exception, with_traceback=exc_info[2], from_=e
File "/home/airflow/.local/lib/python3.7/site-packages/sqlalchemy/util/compat.py", line 207, in raise_
raise exception
File "/home/airflow/.local/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 1803, in _execute_context
cursor, statement, parameters, context
File "/home/airflow/.local/lib/python3.7/site-packages/sqlalchemy/engine/default.py", line 719, in do_execute
cursor.execute(statement, parameters)
sqlalchemy.exc.DatabaseError: (psycopg2.DatabaseError) could not receive data from server: Connection timed out
SSL SYSCALL error: Connection timed out
[SQL: SELECT celery_taskmeta.id AS celery_taskmeta_id, celery_taskmeta.task_id AS celery_taskmeta_task_id, celery_taskmeta.status AS celery_taskmeta_status, celery_taskmeta.result AS celery_taskmeta_result, celery_taskmeta.date_done AS celery_taskmeta_date_done, celery_taskmeta.traceback AS celery_taskmeta_traceback
FROM celery_taskmeta
WHERE celery_taskmeta.task_id = %(task_id_1)s]
[parameters: {'task_id_1': 'c5f9f53c-8afe-4d67-8d3b-d7ad84875de1'}]
(Background on this error at: https://sqlalche.me/e/14/4xp6)
[2022-09-22 18:55:50,929: INFO/ForkPoolWorker-15] [c5f9f53c-8afe-4d67-8d3b-d7ad84875de1] Executing command in Celery: ['airflow', 'tasks', 'run', 'CS_ALERTING', 'CheckRunningTasks', 'scheduled__2022-09-22T18:00:00+00:00', '--local', '--subdir', 'DAGS_FOLDER/CS_ALERTING.py']
[2022-09-22 18:55:51,241: INFO/ForkPoolWorker-15] Filling up the DagBag from /opt/airflow/platform_pam/dags/CS_ALERTING.py
[2022-09-22 18:55:53,467: INFO/ForkPoolWorker-15] Running <TaskInstance: CS_ALERTING.CheckRunningTasks scheduled__2022-09-22T18:00:00+00:00 [queued]> on host localhost
[2022-09-22 18:55:58,304: INFO/ForkPoolWorker-15] Task airflow.executors.celery_executor.execute_command[c5f9f53c-8afe-4d67-8d3b-d7ad84875de1] succeeded in 960.1964174450004s: None
[2022-09-22 19:29:25,931: WARNING/MainProcess] consumer: Connection to broker lost. Trying to re-establish the connection...
Traceback (most recent call last):
File "/home/airflow/.local/lib/python3.7/site-packages/redis/connection.py", line 706, in send_packed_command
sendall(self._sock, item)
File "/home/airflow/.local/lib/python3.7/site-packages/redis/_compat.py", line 9, in sendall
return sock.sendall(*args, **kwargs)
File "/usr/local/lib/python3.7/ssl.py", line 1034, in sendall
v = self.send(byte_view[count:])
File "/usr/local/lib/python3.7/ssl.py", line 1003, in send
return self._sslobj.write(data)
TimeoutError: [Errno 110] Connection timed out
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/airflow/.local/lib/python3.7/site-packages/celery/worker/consumer/consumer.py", line 332, in start
blueprint.start(self)
File "/home/airflow/.local/lib/python3.7/site-packages/celery/bootsteps.py", line 116, in start
step.start(parent)
File "/home/airflow/.local/lib/python3.7/site-packages/celery/worker/consumer/consumer.py", line 628, in start
c.loop(*c.loop_args())
File "/home/airflow/.local/lib/python3.7/site-packages/celery/worker/loops.py", line 97, in asynloop
next(loop)
File "/home/airflow/.local/lib/python3.7/site-packages/kombu/asynchronous/hub.py", line 301, in create_loop
poll_timeout = fire_timers(propagate=propagate) if scheduled else 1
File "/home/airflow/.local/lib/python3.7/site-packages/kombu/asynchronous/hub.py", line 143, in fire_timers
entry()
File "/home/airflow/.local/lib/python3.7/site-packages/kombu/asynchronous/timer.py", line 64, in __call__
return self.fun(*self.args, **self.kwargs)
File "/home/airflow/.local/lib/python3.7/site-packages/kombu/asynchronous/timer.py", line 126, in _reschedules
return fun(*args, **kwargs)
File "/home/airflow/.local/lib/python3.7/site-packages/kombu/transport/redis.py", line 557, in maybe_check_subclient_health
client.check_health()
File "/home/airflow/.local/lib/python3.7/site-packages/redis/client.py", line 3522, in check_health
check_health=False)
File "/home/airflow/.local/lib/python3.7/site-packages/redis/connection.py", line 726, in send_command
check_health=kwargs.get('check_health', True))
File "/home/airflow/.local/lib/python3.7/site-packages/redis/connection.py", line 718, in send_packed_command
(errno, errmsg))
redis.exceptions.ConnectionError: Error 110 while writing to socket. Connection timed out.
[2022-09-22 19:29:26,023: WARNING/MainProcess] /home/airflow/.local/lib/python3.7/site-packages/celery/worker/consumer/consumer.py:367: CPendingDeprecationWarning:
I referred the Airflow's documentation and found setting up database
We are modifying Airflow's docker image and adding a python file,
airflow.www.db_utils.db_config (This file is installed to site_packages) and defined the dictionary
keepalive_kwargs = {
"keepalives": 1,
"keepalives_idle": 30,
"keepalives_interval": 5,
"keepalives_count": 5,
}
Finally, we are setting
ENV AIRFLOW__DATABASE__SQL_ALCHEMY_CONNECT_ARGS="airflow.www.db_utils.db_config.keepalive_kwargs"
Unfortunately, the error stills persist. It will be great if someone helps me to resolve this issue.
Good morning,
i am using tensorflow lite and i also wanted to use telepot.
I also installed the Coral USB accelerator, but I don't think it depends on him, also because it is independent of whether or not to add --edgetpu to the end of the program start command.
It works only if the transfer of messages or images I place it before this instruction:
from tensorflow.lite.python.interpreter import Interpreter
it's how telepot is incompatible with tflite.
Obviously everything works without the telepot instruction
What can I do?
I'm using a raspberry pi 4 with s.o. Debian burst, Python 3.7 and Opencv 4.1
This is the error that gives me:
Traceback (most recent call last):
File "/home/pi/tflite1/tflite1-env/lib/python3.7/site-packages/urllib3/contrib/pyopenssl.py", line 485, in wrap_socket
cnx.do_handshake()
File "/usr/lib/python3/dist-packages/OpenSSL/SSL.py", line 1915, in do_handshake
self._raise_ssl_error(self._ssl, result)
File "/usr/lib/python3/dist-packages/OpenSSL/SSL.py", line 1647, in _raise_ssl_error
_raise_current_error()
File "/usr/lib/python3/dist-packages/OpenSSL/_util.py", line 54, in exception_from_error_queue
raise exception_type(errors)
OpenSSL.SSL.Error: [('SSL routines', 'tls_process_server_certificate', 'certificate verify failed')]
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/pi/tflite1/tflite1-env/lib/python3.7/site-packages/urllib3/connectionpool.py", line 672, in urlopen
chunked=chunked,
File "/home/pi/tflite1/tflite1-env/lib/python3.7/site-packages/urllib3/connectionpool.py", line 376, in _make_request
self._validate_conn(conn)
File "/home/pi/tflite1/tflite1-env/lib/python3.7/site-packages/urllib3/connectionpool.py", line 994, in _validate_conn
conn.connect()
File "/home/pi/tflite1/tflite1-env/lib/python3.7/site-packages/urllib3/connection.py", line 394, in connect
ssl_context=context,
File "/home/pi/tflite1/tflite1-env/lib/python3.7/site-packages/urllib3/util/ssl_.py", line 370, in ssl_wrap_socket
return context.wrap_socket(sock, server_hostname=server_hostname)
File "/home/pi/tflite1/tflite1-env/lib/python3.7/site-packages/urllib3/contrib/pyopenssl.py", line 491, in wrap_socket
raise ssl.SSLError("bad handshake: %r" % e)
ssl.SSLError: ("bad handshake: Error([('SSL routines', 'tls_process_server_certificate', 'certificate verify failed')])",)
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "TFLite_detection_webcam_Prova1.py", line 137, in <module>
spedisci()
File "TFLite_detection_webcam_Prova1.py", line 33, in spedisci
bot.sendPhoto(256868258, foto)
File "/usr/local/lib/python3.7/dist-packages/telepot/__init__.py", line 539, in sendPhoto
return self._api_request_with_file('sendPhoto', _rectify(p), 'photo', photo)
File "/usr/local/lib/python3.7/dist-packages/telepot/__init__.py", line 499, in _api_request_with_file
return self._api_request(method, _rectify(params), files, **kwargs)
File "/usr/local/lib/python3.7/dist-packages/telepot/__init__.py", line 491, in _api_request
return api.request((self._token, method, params, files), **kwargs)
File "/usr/local/lib/python3.7/dist-packages/telepot/api.py", line 154, in request
r = fn(*args, **kwargs) # `fn` must be thread-safe
File "/home/pi/tflite1/tflite1-env/lib/python3.7/site-packages/urllib3/request.py", line 171, in request_encode_body
return self.urlopen(method, url, **extra_kw)
File "/home/pi/tflite1/tflite1-env/lib/python3.7/site-packages/urllib3/poolmanager.py", line 330, in urlopen
response = conn.urlopen(method, u.request_uri, **kw)
File "/home/pi/tflite1/tflite1-env/lib/python3.7/site-packages/urllib3/connectionpool.py", line 760, in urlopen
**response_kw
File "/home/pi/tflite1/tflite1-env/lib/python3.7/site-packages/urllib3/connectionpool.py", line 760, in urlopen
**response_kw
File "/home/pi/tflite1/tflite1-env/lib/python3.7/site-packages/urllib3/connectionpool.py", line 760, in urlopen
**response_kw
File "/home/pi/tflite1/tflite1-env/lib/python3.7/site-packages/urllib3/connectionpool.py", line 720, in urlopen
method, url, error=e, _pool=self, _stacktrace=sys.exc_info()[2]
File "/home/pi/tflite1/tflite1-env/lib/python3.7/site-packages/urllib3/util/retry.py", line 436, in increment
raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool
(host='api.telegram.org', port=443): Max retries exceeded with url: /bot926377239:AAEl0gqMWzG0dMidkGNqcGr2wkeTLbgZn3g/sendPhoto (Caused by SSLError(SSLError("bad handshake: Error([('SSL routines', 'tls_process_server_certificate', 'certificate verify failed')])")))
I am trying to use the responses module to mock an HTTP server and I have the feeling it is broken when calls requests are multi-threaded.
Consider the following:
import json
import responses
import requests
import threading
try:
import urlparse
except ModuleNotFoundError:
import urllib.parse as urlparse
server_url = 'http://server'
headers_json = {'content-type': 'application/json'}
def init():
def endpoint(request):
id = urlparse.parse_qs(urlparse.urlparse(request.path_url).query)["id"][0]
data = {"foo": int(id)}
return 200, headers_json, json.dumps(data)
responses.add_callback(
responses.GET, server_url + '/',
callback=endpoint,
)
#responses.activate
def test():
init()
def responses_routine():
resp = requests.get(
server_url + '/?id=456',
headers=headers_json,
)
# {"foo": "456"}
print(resp.json()["foo"])
def print_routine():
print("456")
# This will print "456"
#responses_routine()
# This will print "456" after 2 seconds
#threading.Timer(2, print_routine).start()
# This will fail
threading.Timer(2, responses_routine).start()
test()
It fails with a stack that clearly indicates that no call to responses was made. The requests module did try to make an HTTP code to a remote server.
Exception in thread Thread-1:
Traceback (most recent call last):
File "/opt/python/python3.6.6/lib/python3.6/site-packages/urllib3/connection.py", line 159, in _new_conn
(self._dns_host, self.port), self.timeout, **extra_kw)
File "/opt/python/python3.6.6/lib/python3.6/site-packages/urllib3/util/connection.py", line 57, in create_connection
for res in socket.getaddrinfo(host, port, family, socket.SOCK_STREAM):
File "/opt/python/python3.6.6/lib/python3.6/socket.py", line 745, in getaddrinfo
for res in _socket.getaddrinfo(host, port, family, type, proto, flags):
socket.gaierror: [Errno -2] Name or service not known
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/opt/python/python3.6.6/lib/python3.6/site-packages/urllib3/connectionpool.py", line 600, in urlopen
chunked=chunked)
File "/opt/python/python3.6.6/lib/python3.6/site-packages/urllib3/connectionpool.py", line 354, in _make_request
conn.request(method, url, **httplib_request_kw)
File "/opt/python/python3.6.6/lib/python3.6/http/client.py", line 1239, in request
self._send_request(method, url, body, headers, encode_chunked)
File "/opt/python/python3.6.6/lib/python3.6/http/client.py", line 1285, in _send_request
self.endheaders(body, encode_chunked=encode_chunked)
File "/opt/python/python3.6.6/lib/python3.6/http/client.py", line 1234, in endheaders
self._send_output(message_body, encode_chunked=encode_chunked)
File "/opt/python/python3.6.6/lib/python3.6/http/client.py", line 1026, in _send_output
self.send(msg)
File "/opt/python/python3.6.6/lib/python3.6/http/client.py", line 964, in send
self.connect()
File "/opt/python/python3.6.6/lib/python3.6/site-packages/urllib3/connection.py", line 181, in connect
conn = self._new_conn()
File "/opt/python/python3.6.6/lib/python3.6/site-packages/urllib3/connection.py", line 168, in _new_conn
self, "Failed to establish a new connection: %s" % e)
urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPConnection object at 0x2aaab0930438>: Failed to establish a new connection: [Errno -2] Name or service not known
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/opt/python/python3.6.6/lib/python3.6/site-packages/requests/adapters.py", line 449, in send
timeout=timeout
File "/opt/python/python3.6.6/lib/python3.6/site-packages/urllib3/connectionpool.py", line 638, in urlopen
_stacktrace=sys.exc_info()[2])
File "/opt/python/python3.6.6/lib/python3.6/site-packages/urllib3/util/retry.py", line 398, in increment
raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPConnectionPool(host='server', port=80): Max retries exceeded with url: /?id=456 (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x2aaab0930438>: Failed to establish a new connection: [Errno -2] Name or service not known',))
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/opt/python/python3.6.6/lib/python3.6/threading.py", line 916, in _bootstrap_inner
self.run()
File "/opt/python/python3.6.6/lib/python3.6/threading.py", line 1182, in run
self.function(*self.args, **self.kwargs)
File "tmp.py", line 35, in responses_routine
headers=headers_json,
File "/opt/python/python3.6.6/lib/python3.6/site-packages/requests/api.py", line 116, in post
return request('post', url, data=data, json=json, **kwargs)
File "/opt/python/python3.6.6/lib/python3.6/site-packages/requests/api.py", line 60, in request
return session.request(method=method, url=url, **kwargs)
File "/opt/python/python3.6.6/lib/python3.6/site-packages/requests/sessions.py", line 533, in request
resp = self.send(prep, **send_kwargs)
File "/opt/python/python3.6.6/lib/python3.6/site-packages/requests/sessions.py", line 646, in send
r = adapter.send(request, **kwargs)
File "/opt/python/python3.6.6/lib/python3.6/site-packages/requests/adapters.py", line 516, in send
raise ConnectionError(e, request=request)
requests.exceptions.ConnectionError: HTTPConnectionPool(host='server', port=80): Max retries exceeded with url: /?id=456 (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x2aaab0930438>: Failed to establish a new connection: [Errno -2] Name or service not known',))
It fails on Redhat 6.6. I am using Python 3.6.6 (it also fails in 2.7 with a similar stack) with requests 2.22 and responses 0.10.6.
It's like when the main thread exited, all "handles" to responses were lost...
Any clues ?
Actually what's created within the context manager responses.activate should not escape it.
In the code above, the underlying thread of the Timer object would be joined outside the scope, hence potentially run outside the context manager scope, which yield the exception.
I am new to FIWARE. On my Ubuntu 16.04, I have setup Fiware's KeyRock which is a combination of OpenStack Keystone Identity Management and Horizon Dashboard. I have used this guide to set up everything.
After installing Keystone (which is a RESTful API), I have used curl to send HTTP requests and everything is working fine, i.e. I can run these commands on the keystone server using curl.
However, when I run Horizon on the same server using the reference Django project located in the openstack_dashboard directory with:
sudo tools/with_venv.sh python manage.py runserver 0.0.0.0:8000,
Horizon server starts with no errors but when I access it via browser, I get the following error:
A server error occured. Please contact the administrator
On the Keystone server console, I get the following error:
2017-01-09 11:53:40.962 13285 ERROR keystone.notifications [-] Failed to construct notifier
2017-01-09 11:53:40.962 13285 TRACE keystone.notifications Traceback (most recent call last):
2017-01-09 11:53:40.962 13285 TRACE keystone.notifications File "/keystone/keystone/notifications.py", line 220, in _get_notifier
2017-01-09 11:53:40.962 13285 TRACE keystone.notifications transport = messaging.get_transport(CONF)
2017-01-09 11:53:40.962 13285 TRACE keystone.notifications File "/keystone/.venv/local/lib/python2.7/site-packages/oslo/messaging/transport.py", line 185, in get_transport
2017-01-09 11:53:40.962 13285 TRACE keystone.notifications invoke_kwds=kwargs)
2017-01-09 11:53:40.962 13285 TRACE keystone.notifications File "/keystone/.venv/local/lib/python2.7/site-packages/stevedore/driver.py", line 45, in __init__
2017-01-09 11:53:40.962 13285 TRACE keystone.notifications verify_requirements=verify_requirements,
2017-01-09 11:53:40.962 13285 TRACE keystone.notifications File "/keystone/.venv/local/lib/python2.7/site-packages/stevedore/named.py", line 55, in __init__
2017-01-09 11:53:40.962 13285 TRACE keystone.notifications verify_requirements)
2017-01-09 11:53:40.962 13285 TRACE keystone.notifications File "/keystone/.venv/local/lib/python2.7/site-packages/stevedore/extension.py", line 170, in _load_plugins
2017-01-09 11:53:40.962 13285 TRACE keystone.notifications self._on_load_failure_callback(self, ep, err)
2017-01-09 11:53:40.962 13285 TRACE keystone.notifications File "/keystone/.venv/local/lib/python2.7/site-packages/stevedore/driver.py", line 50, in _default_on_load_failure
2017-01-09 11:53:40.962 13285 TRACE keystone.notifications raise err
2017-01-09 11:53:40.962 13285 TRACE keystone.notifications ImportError: cannot import name _uuid_generate_random
2017-01-09 11:53:40.962 13285 TRACE keystone.notifications
2017-01-09 11:53:40.964 13285 WARNING keystone.common.wsgi [-] Authorization failed. The request you have made requires authentication. from 127.0.0.1
2017-01-09 11:53:40.970 13285 INFO eventlet.wsgi.server [-] 127.0.0.1 - - [09/Jan/2017 11:53:40] "POST /v3/auth/tokens HTTP/1.1" 401 331 0.261293
On the Horizon server console, I get the following error:
DEBUG:idm_logger:Creating a new internal keystoneclient connection to http://0.0.0.0:5000/v3.
Unauthorized: The request you have made requires authentication. (HTTP 401)
Traceback (most recent call last):
File "/horizon/openstack_dashboard/fiware_api/keystone.py", line 990, in _get_element_and_cache
resource_element = function(request, element)
File "/horizon/openstack_dashboard/fiware_api/keystone.py", line 1022, in <lambda>
request, basic, lambda req, n: internal_keystoneclient(req).roles.find(name=n), pickle_props=['name'])
File "/horizon/openstack_dashboard/fiware_api/keystone.py", line 64, in internal_keystoneclient
cache.set(CACHE_CLIENT, keystoneclient.session.get_token(), INTERNAL_CLIENT_CACHE_TIME)
File "/horizon/.venv/src/python-keystoneclient/keystoneclient/session.py", line 610, in get_token
return (self.get_auth_headers(auth) or {}).get('X-Auth-Token')
File "/horizon/.venv/src/python-keystoneclient/keystoneclient/session.py", line 589, in get_auth_headers
return auth.get_headers(self, **kwargs)
File "/horizon/.venv/src/python-keystoneclient/keystoneclient/auth/base.py", line 114, in get_headers
token = self.get_token(session)
File "/horizon/.venv/src/python-keystoneclient/keystoneclient/auth/identity/base.py", line 104, in get_token
return self.get_access(session).auth_token
File "/horizon/.venv/src/python-keystoneclient/keystoneclient/auth/identity/base.py", line 144, in get_access
self.auth_ref = self.get_auth_ref(session)
File "/horizon/.venv/src/python-keystoneclient/keystoneclient/auth/identity/v3.py", line 127, in get_auth_ref
authenticated=False, log=False, **rkwargs)
File "/horizon/.venv/src/python-keystoneclient/keystoneclient/session.py", line 488, in post
return self.request(url, 'POST', **kwargs)
File "/horizon/.venv/src/python-keystoneclient/keystoneclient/utils.py", line 318, in inner
return func(*args, **kwargs)
File "/horizon/.venv/src/python-keystoneclient/keystoneclient/session.py", line 389, in request
raise exceptions.from_response(resp, method, url)
Unauthorized: The request you have made requires authentication. (HTTP 401)
Traceback (most recent call last):
File "/usr/lib/python2.7/wsgiref/handlers.py", line 85, in run
self.result = application(self.environ, self.start_response)
File "/horizon/.venv/local/lib/python2.7/site-packages/django/contrib/staticfiles/handlers.py", line 64, in __call__
return self.application(environ, start_response)
File "/horizon/.venv/local/lib/python2.7/site-packages/django/core/handlers/wsgi.py", line 168, in __call__
self.load_middleware()
File "/horizon/.venv/local/lib/python2.7/site-packages/django/core/handlers/base.py", line 46, in load_middleware
mw_instance = mw_class()
File "/horizon/.venv/local/lib/python2.7/site-packages/django/middleware/locale.py", line 23, in __init__
for url_pattern in get_resolver(None).url_patterns:
File "/horizon/.venv/local/lib/python2.7/site-packages/django/core/urlresolvers.py", line 367, in url_patterns
patterns = getattr(self.urlconf_module, "urlpatterns", self.urlconf_module)
File "/horizon/.venv/local/lib/python2.7/site-packages/django/core/urlresolvers.py", line 361, in urlconf_module
self._urlconf_module = import_module(self.urlconf_name)
File "/usr/lib/python2.7/importlib/__init__.py", line 37, in import_module
__import__(name)
File "/horizon/openstack_dashboard/urls.py", line 36, in <module>
from openstack_dashboard.dashboards.idm_admin.user_accounts \
File "/horizon/openstack_dashboard/dashboards/idm_admin/user_accounts/views.py", line 28, in <module>
from openstack_dashboard.dashboards.idm_admin.user_accounts \
File "/horizon/openstack_dashboard/dashboards/idm_admin/user_accounts/forms.py", line 195, in <module>
class UpdateAccountForm(forms.SelfHandlingForm, UserAccountsLogicMixin):
File "/horizon/openstack_dashboard/dashboards/idm_admin/user_accounts/forms.py", line 202, in UpdateAccountForm
choices=get_account_choices())
File "/horizon/openstack_dashboard/dashboards/idm_admin/user_accounts/forms.py", line 172, in get_account_choices
use_idm_account=True),
File "/horizon/openstack_dashboard/fiware_api/keystone.py", line 1022, in get_basic_role
request, basic, lambda req, n: internal_keystoneclient(req).roles.find(name=n), pickle_props=['name'])
File "/horizon/openstack_dashboard/fiware_api/keystone.py", line 997, in _get_element_and_cache
exceptions.handle(request)
File "/horizon/horizon/exceptions.py", line 291, in handle
messages.error(request, message or fallback)
File "/horizon/horizon/messages.py", line 83, in error
fail_silently=fail_silently)
File "/horizon/horizon/messages.py", line 41, in add_message
if not horizon_message_already_queued(request, message):
File "/horizon/horizon/messages.py", line 28, in horizon_message_already_queued
if request.is_ajax():
AttributeError: 'NoneType' object has no attribute 'is_ajax'
[09/Jan/2017 12:39:35] "GET / HTTP/1.1" 500 59
I am sending a GET request via the browser but the Keystone server is receiving a POST request as indicated in its console output. I don't understand why is this so.
It looks like the Horizon server is not able to verify the credentials.
In the case of Fiware, you would get this error on Horizon server if the file local_settings.py (located in the /horizon/openstack_dashboard/local/ directory) is not properly configured. In this file, there is a dictionary named IDM_USER_CREDENTIALS where you have to set the password for the admin profile that was created when you populated the database of Keystone. It is also given in the link that you provided. Try changing the password and see if it works.
I'm running a Flask app via uWSGI which is managed by Supervisor (and the webserver is using Nginx).
The whole setup seems to work perfectly, however, I notice that every day or two, my flask app starts throwing 500 errors (which means there is a Sqlalchemy db.commit() error) and I can't recover until I go in and reload uwsgi, after which, everything works perfectly again. To reload, I typically kill the uwsgi processes, and supervisor brings them back in and it's fine. Once I've gotten it to work by just 'touch'ing the wsgi file.
Also strangely, whenever I run supervisor with processes=1, it automatically loads TWO uwsgi instances. The first PID is the one logged as being operated by supervisor, and the other doesn't register anywhere. However, regardless of this, killing the 2nd process - it always comes back.
Below is my uwsgi startup command run by supervisor
[program:{{ app_name }}]
command=/usr/local/bin/uwsgi
--socket=/tmp/{{ app_name }}.sock
--logto={{ log_dir }}/application.log
--home={{ site_dir }}/env
--pythonpath={{ site_dir }}
--virtualenv={{ env_path }}
--wsgi-file={{ webapps_dir }}/manage.py
--callable={{ wsgi_callable }}
--max-requests=1000
--master
--processes=1
--uid={{ deployment_user }}
--gid=www-data
--chmod-socket=664
--chown-socket={{ deployment_user }}:www-data
directory={{ webapps_dir }}
autostart=true
autorestart=true
EDIT: Here is the error that's spit out from the server.
2015-03-19 15:03:38,389 ERROR: Exception on /api/v1/account/login [POST] [in /var/www/website.com/env/local/lib/python2.7/site-packages/flask/app.py:1423]
Traceback (most recent call last):
File "/var/www/website.com/env/local/lib/python2.7/site-packages/flask/app.py", line 1817, in wsgi_app
response = self.full_dispatch_request()
File "/var/www/website.com/env/local/lib/python2.7/site-packages/flask/app.py", line 1477, in full_dispatch_request
rv = self.handle_user_exception(e)
File "/var/www/website.com/env/local/lib/python2.7/site-packages/flask_cors.py", line 272, in wrapped_function
return cors_after_request(app.make_response(f(*args, **kwargs)))
File "/var/www/website.com/env/local/lib/python2.7/site-packages/flask/app.py", line 1381, in handle_user_exception
reraise(exc_type, exc_value, tb)
File "/var/www/website.com/env/local/lib/python2.7/site-packages/flask/app.py", line 1475, in full_dispatch_request
rv = self.dispatch_request()
File "/var/www/website.com/env/local/lib/python2.7/site-packages/flask/app.py", line 1461, in dispatch_request
return self.view_functions[rule.endpoint](**req.view_args)
File "/var/www/website.com/env/local/lib/python2.7/site-packages/flask_classy.py", line 200, in proxy
response = view(**request.view_args)
File "./webportal/account/api.py", line 34, in login_user
user = auth.authenticate(username=email, password=password)
File "/var/www/website.com/server/webportal/auth.py", line 23, in authenticate
user = User.query.filter_by(email=username).first()
File "/var/www/website.com/env/local/lib/python2.7/site-packages/sqlalchemy/orm/query.py", line 2367, in first
ret = list(self[0:1])
File "/var/www/website.com/env/local/lib/python2.7/site-packages/sqlalchemy/orm/query.py", line 2228, in __getitem__
return list(res)
File "/var/www/website.com/env/local/lib/python2.7/site-packages/sqlalchemy/orm/query.py", line 2438, in __iter__
return self._execute_and_instances(context)
File "/var/www/website.com/env/local/lib/python2.7/site-packages/sqlalchemy/orm/query.py", line 2453, in _execute_and_instances
result = conn.execute(querycontext.statement, self._params)
File "/var/www/website.com/env/local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 729, in execute
return meth(self, multiparams, params)
File "/var/www/website.com/env/local/lib/python2.7/site-packages/sqlalchemy/sql/elements.py", line 322, in _execute_on_connection
return connection._execute_clauseelement(self, multiparams, params)
File "/var/www/website.com/env/local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 826, in _execute_clauseelement
compiled_sql, distilled_params
File "/var/www/website.com/env/local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 958, in _execute_context
context)
File "/var/www/website.com/env/local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1159, in _handle_dbapi_exception
exc_info
File "/var/www/website.com/env/local/lib/python2.7/site-packages/sqlalchemy/util/compat.py", line 199, in raise_from_cause
reraise(type(exception), exception, tb=exc_tb)
File "/var/www/website.com/env/local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 951, in _execute_context
context)
File "/var/www/website.com/env/local/lib/python2.7/site-packages/sqlalchemy/engine/default.py", line 436, in do_execute
cursor.execute(statement, parameters)
File "/var/www/website.com/env/local/lib/python2.7/site-packages/MySQLdb/cursors.py", line 205, in execute
self.errorhandler(self, exc, value)
File "/var/www/website.com/env/local/lib/python2.7/site-packages/MySQLdb/connections.py", line 36, in defaulterrorhandler
raise errorclass, errorvalue
OperationalError: (OperationalError) (2006, 'MySQL server has gone away') 'SELECT `Users`.id AS `Users_id`, `Users`.email AS `Users_email`, `Users`.password AS `Users_password`, `Users`.`emailConfirmed` AS `Users_emailConfirmed`, `Users`.`createdAtUtc` AS `Users_createdAtUtc`, `Users`.`updatedAtUtc` AS `Users_updatedAtUtc`, `Users`.`roleId` AS `Users_roleId` \nFROM `Users` \nWHERE `Users`.email = %s \n LIMIT %s' ('user#website.demo', 1)
2015-03-19 15:06:22,089 ERROR: Exception on /api/v1/account/login [POST] [in /var/www/website.com/env/local/lib/python2.7/site-packages/flask/app.py:1423]
Traceback (most recent call last):
File "/var/www/website.com/env/local/lib/python2.7/site-packages/flask/app.py", line 1817, in wsgi_app
response = self.full_dispatch_request()
File "/var/www/website.com/env/local/lib/python2.7/site-packages/flask/app.py", line 1477, in full_dispatch_request
rv = self.handle_user_exception(e)
File "/var/www/website.com/env/local/lib/python2.7/site-packages/flask_cors.py", line 272, in wrapped_function
return cors_after_request(app.make_response(f(*args, **kwargs)))
File "/var/www/website.com/env/local/lib/python2.7/site-packages/flask/app.py", line 1381, in handle_user_exception
reraise(exc_type, exc_value, tb)
File "/var/www/website.com/env/local/lib/python2.7/site-packages/flask/app.py", line 1475, in full_dispatch_request
rv = self.dispatch_request()
File "/var/www/website.com/env/local/lib/python2.7/site-packages/flask/app.py", line 1461, in dispatch_request
return self.view_functions[rule.endpoint](**req.view_args)
File "/var/www/website.com/env/local/lib/python2.7/site-packages/flask_classy.py", line 200, in proxy
response = view(**request.view_args)
File "./webportal/account/api.py", line 34, in login_user
user = auth.authenticate(username=email, password=password)
File "/var/www/website.com/server/webportal/auth.py", line 23, in authenticate
user = User.query.filter_by(email=username).first()
File "/var/www/website.com/env/local/lib/python2.7/site-packages/sqlalchemy/orm/query.py", line 2367, in first
ret = list(self[0:1])
File "/var/www/website.com/env/local/lib/python2.7/site-packages/sqlalchemy/orm/query.py", line 2228, in __getitem__
return list(res)
File "/var/www/website.com/env/local/lib/python2.7/site-packages/sqlalchemy/orm/query.py", line 2438, in __iter__
return self._execute_and_instances(context)
File "/var/www/website.com/env/local/lib/python2.7/site-packages/sqlalchemy/orm/query.py", line 2453, in _execute_and_instances
result = conn.execute(querycontext.statement, self._params)
File "/var/www/website.com/env/local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 729, in execute
return meth(self, multiparams, params)
File "/var/www/website.com/env/local/lib/python2.7/site-packages/sqlalchemy/sql/elements.py", line 322, in _execute_on_connection
return connection._execute_clauseelement(self, multiparams, params)
File "/var/www/website.com/env/local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 826, in _execute_clauseelement
compiled_sql, distilled_params
File "/var/www/website.com/env/local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 893, in _execute_context
None, None)
File "/var/www/website.com/env/local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1159, in _handle_dbapi_exception
exc_info
File "/var/www/website.com/env/local/lib/python2.7/site-packages/sqlalchemy/util/compat.py", line 199, in raise_from_cause
reraise(type(exception), exception, tb=exc_tb)
File "/var/www/website.com/env/local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 887, in _execute_context
conn = self._revalidate_connection()
File "/var/www/website.com/env/local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 242, in _revalidate_connection
"Can't reconnect until invalid "
StatementError: Can't reconnect until invalid transaction is rolled back (original cause: InvalidRequestError: Can't reconnect until invalid transaction is rolled back) u'SELECT `Users`.id AS `Users_id`, `Users`.email AS `Users_email`, `Users`.password AS `Users_password`, `Users`.`emailConfirmed` AS `Users_emailConfirmed`, `Users`.`createdAtUtc` AS `Users_createdAtUtc`, `Users`.`updatedAtUtc` AS `Users_updatedAtUtc`, `Users`.`roleId` AS `Users_roleId` \nFROM `Users` \nWHERE `Users`.email = %s \n LIMIT %s' [immutabledict({})]
According to https://dev.mysql.com/doc/refman/5.0/en/gone-away.html, MySQL closes the connection after 8 hours... Once that happens, the app gets in a bit of a state.