OperationalError: attempt to write a readonly DB on Vercel via Github (the DB and its folder are full-access on my local comp) - sqlite

I am trying to deploy a simple Flask app with a SQLite database. It works fine when running locally. I deploy it on Vercel via Github. It works until it needs to write to the db. Then there is
OperationalError: attempt to write a readonly database
Both the DB and the folder ‘instance’ that contains it are Full-access on my local system, I can see it with
attrib ~/instance
in the command line (I am working with Windows 10). Is there a way to check and change the access mode of a file or a folder right on the Github repo?
Here are the logs regarding this error:
[ERROR] 2023-01-12T17:19:33.931Z 63bacb5b-ec13-4247-b784-9aaba99376a0
Exception on /register [POST]Traceback (most recent call last):
File "/var/task/sqlalchemy/engine/base.py", line 1900, in _execute_context self.dialect.do_execute
( File "/var/task/sqlalchemy/engine/default.py", line 736, in do_execute cursor.execute(statement, parameters)
sqlite3.OperationalError: attempt to write a readonly database
The above exception was the direct cause of the following exception:
Traceback (most recent call last): File "/var/task/flask/app.py", line 2525, in wsgi_app response = self.full_dispatch_request() File "/var/task/flask/app.py", line 1822, in full_dispatch_request rv = self.handle_user_exception(e) File "/var/task/flask/app.py", line 1820, in full_dispatch_request rv = self.dispatch_request() File "/var/task/flask/app.py", line 1796, in dispatch_request return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) File "./app.py", line 136, in register db.session.commit()
File "<string>", line 2, in commit File "/var/task/sqlalchemy/orm/session.py", line 1451, in commit self._transaction.commit(_to_root=self.future) File "/var/task/sqlalchemy/orm/session.py", line 829, in commit self._prepare_impl() File "/var/task/sqlalchemy/orm/session.py", line 808, in _prepare_impl self.session.flush() File "/var/task/sqlalchemy/orm/session.py", line 3444, in flush self._flush(objects) File "/var/task/sqlalchemy/orm/session.py", line 3584, in _flush transaction.rollback(_capture_exception=True) File "/var/task/sqlalchemy/util/langhelpers.py", line 70, in __exit__ compat.raise_( File "/var/task/sqlalchemy/util/compat.py", line 211, in raise_ raise exception File "/var/task/sqlalchemy/orm/session.py", line 3544, in _flush flush_context.execute() File "/var/task/sqlalchemy/orm/unitofwork.py", line 456, in execute rec.execute(self) File "/var/task/sqlalchemy/orm/unitofwork.py", line 630, in execute util.preloaded.orm_persistence.save_obj( File "/var/task/sqlalchemy/orm/persistence.py", line 245, in save_obj _emit_insert_statements( File "/var/task/sqlalchemy/orm/persistence.py", line 1238, in _emit_insert_statements result = connection._execute_20( File "/var/task/sqlalchemy/engine/base.py", line 1705, in _execute_20 return meth(self, args_10style, kwargs_10style, execution_options) File "/var/task/sqlalchemy/sql/elements.py", line 334, in _execute_on_connection return connection._execute_clauseelement( File "/var/task/sqlalchemy/engine/base.py", line 1572, in _execute_clauseelement ret = self._execute_context( File "/var/task/sqlalchemy/engine/base.py", line 1943, in _execute_context self._handle_dbapi_exception( File "/var/task/sqlalchemy/engine/base.py", line 2124, in _handle_dbapi_exception util.raise_( File "/var/task/sqlalchemy/util/compat.py", line 211, in raise_ raise exception File "/var/task/sqlalchemy/engine/base.py", line 1900, in _execute_context self.dialect.do_execute( File "/var/task/sqlalchemy/engine/default.py", line 736, in do_execute cursor.execute(statement, parameters)sqlalchemy.exc.
OperationalError: (sqlite3.OperationalError) attempt to write a readonly database
I double-checked the access mode of the DB in my local system both as User and as Admin

It turns out that Vercel does not support SQLite. It is clearly written in their guides:
SQLite needs a local file system on the server to store the data
permanently when write requests are made. In a serverless environment,
this central single permanent storage is not available because storage
is ephemeral with serverless functions.
So it's not the problem of a Read-only mode, but rather the issue with SQL databases on Vercel.

Related

Apache Airflow on Azure sqlalchemy Postgres SSL SYSCALL error: EOF detected

We deployed Apache Airflow 2.3.3 to Azure.
Webserver - Web App
Scheduler - ACI
Celery Worker - ACI
We were seeing errors on the Celery ACI console related to Postgres and Redis connection timeouts as below
[2022-09-22 18:55:50,650: WARNING/ForkPoolWorker-15] Failed operation _store_result. Retrying 2 more times.
Traceback (most recent call last):
File "/home/airflow/.local/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 1803, in _execute_context
cursor, statement, parameters, context
File "/home/airflow/.local/lib/python3.7/site-packages/sqlalchemy/engine/default.py", line 719, in do_execute
cursor.execute(statement, parameters)
psycopg2.DatabaseError: could not receive data from server: Connection timed out
SSL SYSCALL error: Connection timed out
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/home/airflow/.local/lib/python3.7/site-packages/celery/backends/database/__init__.py", line 47, in _inner
return fun(*args, **kwargs)
File "/home/airflow/.local/lib/python3.7/site-packages/celery/backends/database/__init__.py", line 117, in _store_result
task = list(session.query(self.task_cls).filter(self.task_cls.task_id == task_id))
File "/home/airflow/.local/lib/python3.7/site-packages/sqlalchemy/orm/query.py", line 2887, in __iter__
return self._iter().__iter__()
File "/home/airflow/.local/lib/python3.7/site-packages/sqlalchemy/orm/query.py", line 2897, in _iter
execution_options={"_sa_orm_load_options": self.load_options},
File "/home/airflow/.local/lib/python3.7/site-packages/sqlalchemy/orm/session.py", line 1689, in execute
result = conn._execute_20(statement, params or {}, execution_options)
File "/home/airflow/.local/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 1614, in _execute_20
return meth(self, args_10style, kwargs_10style, execution_options)
File "/home/airflow/.local/lib/python3.7/site-packages/sqlalchemy/sql/elements.py", line 326, in _execute_on_connection
self, multiparams, params, execution_options
File "/home/airflow/.local/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 1491, in _execute_clauseelement
cache_hit=cache_hit,
File "/home/airflow/.local/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 1846, in _execute_context
e, statement, parameters, cursor, context
File "/home/airflow/.local/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 2027, in _handle_dbapi_exception
sqlalchemy_exception, with_traceback=exc_info[2], from_=e
File "/home/airflow/.local/lib/python3.7/site-packages/sqlalchemy/util/compat.py", line 207, in raise_
raise exception
File "/home/airflow/.local/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 1803, in _execute_context
cursor, statement, parameters, context
File "/home/airflow/.local/lib/python3.7/site-packages/sqlalchemy/engine/default.py", line 719, in do_execute
cursor.execute(statement, parameters)
sqlalchemy.exc.DatabaseError: (psycopg2.DatabaseError) could not receive data from server: Connection timed out
SSL SYSCALL error: Connection timed out
[SQL: SELECT celery_taskmeta.id AS celery_taskmeta_id, celery_taskmeta.task_id AS celery_taskmeta_task_id, celery_taskmeta.status AS celery_taskmeta_status, celery_taskmeta.result AS celery_taskmeta_result, celery_taskmeta.date_done AS celery_taskmeta_date_done, celery_taskmeta.traceback AS celery_taskmeta_traceback
FROM celery_taskmeta
WHERE celery_taskmeta.task_id = %(task_id_1)s]
[parameters: {'task_id_1': 'c5f9f53c-8afe-4d67-8d3b-d7ad84875de1'}]
(Background on this error at: https://sqlalche.me/e/14/4xp6)
[2022-09-22 18:55:50,929: INFO/ForkPoolWorker-15] [c5f9f53c-8afe-4d67-8d3b-d7ad84875de1] Executing command in Celery: ['airflow', 'tasks', 'run', 'CS_ALERTING', 'CheckRunningTasks', 'scheduled__2022-09-22T18:00:00+00:00', '--local', '--subdir', 'DAGS_FOLDER/CS_ALERTING.py']
[2022-09-22 18:55:51,241: INFO/ForkPoolWorker-15] Filling up the DagBag from /opt/airflow/platform_pam/dags/CS_ALERTING.py
[2022-09-22 18:55:53,467: INFO/ForkPoolWorker-15] Running <TaskInstance: CS_ALERTING.CheckRunningTasks scheduled__2022-09-22T18:00:00+00:00 [queued]> on host localhost
[2022-09-22 18:55:58,304: INFO/ForkPoolWorker-15] Task airflow.executors.celery_executor.execute_command[c5f9f53c-8afe-4d67-8d3b-d7ad84875de1] succeeded in 960.1964174450004s: None
[2022-09-22 19:29:25,931: WARNING/MainProcess] consumer: Connection to broker lost. Trying to re-establish the connection...
Traceback (most recent call last):
File "/home/airflow/.local/lib/python3.7/site-packages/redis/connection.py", line 706, in send_packed_command
sendall(self._sock, item)
File "/home/airflow/.local/lib/python3.7/site-packages/redis/_compat.py", line 9, in sendall
return sock.sendall(*args, **kwargs)
File "/usr/local/lib/python3.7/ssl.py", line 1034, in sendall
v = self.send(byte_view[count:])
File "/usr/local/lib/python3.7/ssl.py", line 1003, in send
return self._sslobj.write(data)
TimeoutError: [Errno 110] Connection timed out
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/airflow/.local/lib/python3.7/site-packages/celery/worker/consumer/consumer.py", line 332, in start
blueprint.start(self)
File "/home/airflow/.local/lib/python3.7/site-packages/celery/bootsteps.py", line 116, in start
step.start(parent)
File "/home/airflow/.local/lib/python3.7/site-packages/celery/worker/consumer/consumer.py", line 628, in start
c.loop(*c.loop_args())
File "/home/airflow/.local/lib/python3.7/site-packages/celery/worker/loops.py", line 97, in asynloop
next(loop)
File "/home/airflow/.local/lib/python3.7/site-packages/kombu/asynchronous/hub.py", line 301, in create_loop
poll_timeout = fire_timers(propagate=propagate) if scheduled else 1
File "/home/airflow/.local/lib/python3.7/site-packages/kombu/asynchronous/hub.py", line 143, in fire_timers
entry()
File "/home/airflow/.local/lib/python3.7/site-packages/kombu/asynchronous/timer.py", line 64, in __call__
return self.fun(*self.args, **self.kwargs)
File "/home/airflow/.local/lib/python3.7/site-packages/kombu/asynchronous/timer.py", line 126, in _reschedules
return fun(*args, **kwargs)
File "/home/airflow/.local/lib/python3.7/site-packages/kombu/transport/redis.py", line 557, in maybe_check_subclient_health
client.check_health()
File "/home/airflow/.local/lib/python3.7/site-packages/redis/client.py", line 3522, in check_health
check_health=False)
File "/home/airflow/.local/lib/python3.7/site-packages/redis/connection.py", line 726, in send_command
check_health=kwargs.get('check_health', True))
File "/home/airflow/.local/lib/python3.7/site-packages/redis/connection.py", line 718, in send_packed_command
(errno, errmsg))
redis.exceptions.ConnectionError: Error 110 while writing to socket. Connection timed out.
[2022-09-22 19:29:26,023: WARNING/MainProcess] /home/airflow/.local/lib/python3.7/site-packages/celery/worker/consumer/consumer.py:367: CPendingDeprecationWarning:
I referred the Airflow's documentation and found setting up database
We are modifying Airflow's docker image and adding a python file,
airflow.www.db_utils.db_config (This file is installed to site_packages) and defined the dictionary
keepalive_kwargs = {
"keepalives": 1,
"keepalives_idle": 30,
"keepalives_interval": 5,
"keepalives_count": 5,
}
Finally, we are setting
ENV AIRFLOW__DATABASE__SQL_ALCHEMY_CONNECT_ARGS="airflow.www.db_utils.db_config.keepalive_kwargs"
Unfortunately, the error stills persist. It will be great if someone helps me to resolve this issue.

AIRFLOW - "Oops Something bad has happened"

Everything was working just fine with my airflow fresh install, until I got this "Oops something bad has happened" message again. Had it previously when editing the connections.
Traceback (most recent call last):
File "/home/mica/.local/lib/python3.8/site-packages/flask/app.py", line 2447, in wsgi_app
response = self.full_dispatch_request()
File "/home/mica/.local/lib/python3.8/site-packages/flask/app.py", line 1952, in full_dispatch_request
rv = self.handle_user_exception(e)
File "/home/mica/.local/lib/python3.8/site-packages/flask/app.py", line 1821, in handle_user_exception
reraise(exc_type, exc_value, tb)
File "/home/mica/.local/lib/python3.8/site-packages/flask/_compat.py", line 39, in reraise
raise value
File "/home/mica/.local/lib/python3.8/site-packages/flask/app.py", line 1950, in full_dispatch_request
rv = self.dispatch_request()
File "/home/mica/.local/lib/python3.8/site-packages/flask/app.py", line 1936, in dispatch_request
return self.view_functions[rule.endpoint](**req.view_args)
File "/home/mica/.local/lib/python3.8/site-packages/airflow/www/auth.py", line 51, in decorated
return func(*args, **kwargs)
File "/home/mica/.local/lib/python3.8/site-packages/airflow/www/views.py", line 718, in index
paging=wwwutils.generate_pages(
File "/home/mica/.local/lib/python3.8/site-packages/airflow/www/utils.py", line 107, in generate_pages
first_node = Markup(
File "/home/mica/.local/lib/python3.8/site-packages/jinja2/utils.py", line 838, in __new__
warnings.warn(
File "/usr/lib/python3.8/warnings.py", line 109, in _showwarnmsg
sw(msg.message, msg.category, msg.filename, msg.lineno,
File "/home/mica/.local/lib/python3.8/site-packages/airflow/settings.py", line 117, in custom_show_warning
write_console.print(msg, soft_wrap=True)
File "/home/mica/.local/lib/python3.8/site-packages/rich/console.py", line 1642, in print
self._buffer.extend(new_segments)
File "/home/mica/.local/lib/python3.8/site-packages/rich/console.py", line 842, in __exit__
self._exit_buffer()
File "/home/mica/.local/lib/python3.8/site-packages/rich/console.py", line 800, in _exit_buffer
self._check_buffer()
File "/home/mica/.local/lib/python3.8/site-packages/rich/console.py", line 1934, in _check_buffer
self.file.write(text)
OSError: [Errno 5] Input/output error
The problem happened when I was launching airflow from the home. If you navigate to where the application is and then start the webserver, it worked for me.

Unable to install "'en_core_web_sm'"

I have installed the spacy 3.1.2 and trying to install en_core_web_sm on jupyter notebook in Jupyterlab by using
!python3 -m spacy download en_core_web_sm
but it is showing the following error
2021-09-01 07:00:12.028366: W tensorflow/stream_executor/platform/default/dso_loader.cc:55] Could not load dynamic library 'libnvinfer.so.6'; dlerror: libnvinfer.so.6: cannot open shared object file: No such file or directory
2021-09-01 07:00:12.028469: W tensorflow/stream_executor/platform/default/dso_loader.cc:55] Could not load dynamic library 'libnvinfer_plugin.so.6'; dlerror: libnvinfer_plugin.so.6: cannot open shared object file: No such file or directory
2021-09-01 07:00:12.028486: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:30] Cannot dlopen some TensorRT libraries. If you would like to use Nvidia GPU with TensorRT, please make sure the missing libraries mentioned above are installed properly.
Traceback (most recent call last):
File "/opt/conda/lib/python3.7/site-packages/urllib3/contrib/pyopenssl.py", line 488, in wrap_socket
cnx.do_handshake()
File "/opt/conda/lib/python3.7/site-packages/OpenSSL/SSL.py", line 1934, in do_handshake
self._raise_ssl_error(self._ssl, result)
File "/opt/conda/lib/python3.7/site-packages/OpenSSL/SSL.py", line 1671, in _raise_ssl_error
_raise_current_error()
File "/opt/conda/lib/python3.7/site-packages/OpenSSL/_util.py", line 54, in exception_from_error_queue
raise exception_type(errors)
OpenSSL.SSL.Error: [('SSL routines', 'tls_process_server_certificate', 'certificate verify failed')]
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/opt/conda/lib/python3.7/site-packages/urllib3/connectionpool.py", line 667, in urlopen
self._prepare_proxy(conn)
File "/opt/conda/lib/python3.7/site-packages/urllib3/connectionpool.py", line 932, in _prepare_proxy
conn.connect()
File "/opt/conda/lib/python3.7/site-packages/urllib3/connection.py", line 371, in connect
ssl_context=context,
File "/opt/conda/lib/python3.7/site-packages/urllib3/util/ssl_.py", line 384, in ssl_wrap_socket
return context.wrap_socket(sock, server_hostname=server_hostname)
File "/opt/conda/lib/python3.7/site-packages/urllib3/contrib/pyopenssl.py", line 494, in wrap_socket
raise ssl.SSLError("bad handshake: %r" % e)
ssl.SSLError: ("bad handshake: Error([('SSL routines', 'tls_process_server_certificate', 'certificate verify failed')])",)
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/opt/conda/lib/python3.7/site-packages/requests/adapters.py", line 449, in send
timeout=timeout
File "/opt/conda/lib/python3.7/site-packages/urllib3/connectionpool.py", line 727, in urlopen
method, url, error=e, _pool=self, _stacktrace=sys.exc_info()[2]
File "/opt/conda/lib/python3.7/site-packages/urllib3/util/retry.py", line 439, in increment
raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='raw.githubusercontent.com', port=443): Max retries exceeded with url: /explosion/spacy-models/master/compatibility.json (Caused by SSLError(SSLError("bad handshake: Error([('SSL routines', 'tls_process_server_certificate', 'certificate verify failed')])")))
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/opt/conda/lib/python3.7/runpy.py", line 193, in _run_module_as_main
"__main__", mod_spec)
File "/opt/conda/lib/python3.7/runpy.py", line 85, in _run_code
exec(code, run_globals)
File "/home/jovyan/.local/lib/python3.7/site-packages/spacy/__main__.py", line 4, in <module>
setup_cli()
File "/home/jovyan/.local/lib/python3.7/site-packages/spacy/cli/_util.py", line 69, in setup_cli
command(prog_name=COMMAND)
File "/opt/conda/lib/python3.7/site-packages/click/core.py", line 829, in __call__
return self.main(*args, **kwargs)
File "/opt/conda/lib/python3.7/site-packages/click/core.py", line 782, in main
rv = self.invoke(ctx)
File "/opt/conda/lib/python3.7/site-packages/click/core.py", line 1259, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/opt/conda/lib/python3.7/site-packages/click/core.py", line 1066, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "/opt/conda/lib/python3.7/site-packages/click/core.py", line 610, in invoke
return callback(*args, **kwargs)
File "/home/jovyan/.local/lib/python3.7/site-packages/typer/main.py", line 497, in wrapper
return callback(**use_params) # type: ignore
File "/home/jovyan/.local/lib/python3.7/site-packages/spacy/cli/download.py", line 35, in download_cli
download(model, direct, sdist, *ctx.args)
File "/home/jovyan/.local/lib/python3.7/site-packages/spacy/cli/download.py", line 67, in download
compatibility = get_compatibility()
File "/home/jovyan/.local/lib/python3.7/site-packages/spacy/cli/download.py", line 78, in get_compatibility
r = requests.get(about.__compatibility__)
File "/opt/conda/lib/python3.7/site-packages/requests/api.py", line 76, in get
return request('get', url, params=params, **kwargs)
File "/opt/conda/lib/python3.7/site-packages/requests/api.py", line 61, in request
return session.request(method=method, url=url, **kwargs)
File "/opt/conda/lib/python3.7/site-packages/requests/sessions.py", line 530, in request
resp = self.send(prep, **send_kwargs)
File "/opt/conda/lib/python3.7/site-packages/requests/sessions.py", line 643, in send
r = adapter.send(request, **kwargs)
File "/opt/conda/lib/python3.7/site-packages/requests/adapters.py", line 514, in send
raise SSLError(e, request=request)
requests.exceptions.SSLError: HTTPSConnectionPool(host='raw.githubusercontent.com', port=443): Max retries exceeded with url: /explosion/spacy-models/master/compatibility.json (Caused by SSLError(SSLError("bad handshake: Error([('SSL routines', 'tls_process_server_certificate', 'certificate verify failed')])")))
Due to this, I can't use
import spacy
nlp = spacy.load('en_core_web_sm')
Because it is showing error as:
OSError: [E050] Can't find model 'en_core_web_sm'. It doesn't seem to be a Python package or a valid path to a data directory.
I have searched various post and articles related to it but nothing is working for me.
Can anyone tell me how to resolve this error ? It would be very helpful for me.

cannot enable nbextensions in Jupyter Notebook with new Conda Environment on IBM DSX

Trying to install this package:-
https://github.com/lspvic/jupyter_tensorboard
Python 2.7
I cannot enable nbextensions in Jupyter Notebook with new Conda Environment
Here is the stacktrace
Enabling notebook extension jupyter_tensorboard/tree...
Traceback (most recent call last):
File "/opt/conda/envs/DSX-Python27/bin/jupyter-nbextension", line 11, in <module>
sys.exit(main())
File "/opt/conda/envs/DSX-Python27/lib/python2.7/site-packages/jupyter_core/application.py", line 267, in launch_instance
return super(JupyterApp, cls).launch_instance(argv=argv, **kwargs)
File "/opt/conda/envs/DSX-Python27/lib/python2.7/site-packages/traitlets/config/application.py", line 658, in launch_instance
app.start()
File "/opt/conda/envs/DSX-Python27/lib/python2.7/site-packages/notebook/nbextensions.py", line 988, in start
super(NBExtensionApp, self).start()
File "/opt/conda/envs/DSX-Python27/lib/python2.7/site-packages/jupyter_core/application.py", line 256, in start
self.subapp.start()
File "/opt/conda/envs/DSX-Python27/lib/python2.7/site-packages/notebook/nbextensions.py", line 898, in start
self.toggle_nbextension(self.extra_args[0])
File "/opt/conda/envs/DSX-Python27/lib/python2.7/site-packages/notebook/nbextensions.py", line 888, in toggle_nbextension
logger=self.log)
File "/opt/conda/envs/DSX-Python27/lib/python2.7/site-packages/notebook/nbextensions.py", line 413, in enable_nbextension
logger=logger)
File "/opt/conda/envs/DSX-Python27/lib/python2.7/site-packages/notebook/nbextensions.py", line 351, in _set_nbextension_state
cm.update(section, {"load_extensions": {require: state}})
File "/opt/conda/envs/DSX-Python27/lib/python2.7/site-packages/traitlets/config/manager.py", line 87, in update
self.set(section_name, data)
File "/opt/conda/envs/DSX-Python27/lib/python2.7/site-packages/traitlets/config/manager.py", line 71, in set
self.ensure_config_dir_exists()
File "/opt/conda/envs/DSX-Python27/lib/python2.7/site-packages/traitlets/config/manager.py", line 46, in ensure_config_dir_exists
os.makedirs(self.config_dir, 0o755)
File "/opt/conda/envs/DSX-Python27/lib/python2.7/os.py", line 157, in makedirs
mkdir(name, mode)
OSError: [Errno 13] Permission denied: '/opt/ibm/dsx/config/nbconfig'
Looks like permission error, Can it be install/enabled with user flag.

Flask/uWSGI/MariaDB running under Supervisor stops working almost daily

I'm running a Flask app via uWSGI which is managed by Supervisor (and the webserver is using Nginx).
The whole setup seems to work perfectly, however, I notice that every day or two, my flask app starts throwing 500 errors (which means there is a Sqlalchemy db.commit() error) and I can't recover until I go in and reload uwsgi, after which, everything works perfectly again. To reload, I typically kill the uwsgi processes, and supervisor brings them back in and it's fine. Once I've gotten it to work by just 'touch'ing the wsgi file.
Also strangely, whenever I run supervisor with processes=1, it automatically loads TWO uwsgi instances. The first PID is the one logged as being operated by supervisor, and the other doesn't register anywhere. However, regardless of this, killing the 2nd process - it always comes back.
Below is my uwsgi startup command run by supervisor
[program:{{ app_name }}]
command=/usr/local/bin/uwsgi
--socket=/tmp/{{ app_name }}.sock
--logto={{ log_dir }}/application.log
--home={{ site_dir }}/env
--pythonpath={{ site_dir }}
--virtualenv={{ env_path }}
--wsgi-file={{ webapps_dir }}/manage.py
--callable={{ wsgi_callable }}
--max-requests=1000
--master
--processes=1
--uid={{ deployment_user }}
--gid=www-data
--chmod-socket=664
--chown-socket={{ deployment_user }}:www-data
directory={{ webapps_dir }}
autostart=true
autorestart=true
EDIT: Here is the error that's spit out from the server.
2015-03-19 15:03:38,389 ERROR: Exception on /api/v1/account/login [POST] [in /var/www/website.com/env/local/lib/python2.7/site-packages/flask/app.py:1423]
Traceback (most recent call last):
File "/var/www/website.com/env/local/lib/python2.7/site-packages/flask/app.py", line 1817, in wsgi_app
response = self.full_dispatch_request()
File "/var/www/website.com/env/local/lib/python2.7/site-packages/flask/app.py", line 1477, in full_dispatch_request
rv = self.handle_user_exception(e)
File "/var/www/website.com/env/local/lib/python2.7/site-packages/flask_cors.py", line 272, in wrapped_function
return cors_after_request(app.make_response(f(*args, **kwargs)))
File "/var/www/website.com/env/local/lib/python2.7/site-packages/flask/app.py", line 1381, in handle_user_exception
reraise(exc_type, exc_value, tb)
File "/var/www/website.com/env/local/lib/python2.7/site-packages/flask/app.py", line 1475, in full_dispatch_request
rv = self.dispatch_request()
File "/var/www/website.com/env/local/lib/python2.7/site-packages/flask/app.py", line 1461, in dispatch_request
return self.view_functions[rule.endpoint](**req.view_args)
File "/var/www/website.com/env/local/lib/python2.7/site-packages/flask_classy.py", line 200, in proxy
response = view(**request.view_args)
File "./webportal/account/api.py", line 34, in login_user
user = auth.authenticate(username=email, password=password)
File "/var/www/website.com/server/webportal/auth.py", line 23, in authenticate
user = User.query.filter_by(email=username).first()
File "/var/www/website.com/env/local/lib/python2.7/site-packages/sqlalchemy/orm/query.py", line 2367, in first
ret = list(self[0:1])
File "/var/www/website.com/env/local/lib/python2.7/site-packages/sqlalchemy/orm/query.py", line 2228, in __getitem__
return list(res)
File "/var/www/website.com/env/local/lib/python2.7/site-packages/sqlalchemy/orm/query.py", line 2438, in __iter__
return self._execute_and_instances(context)
File "/var/www/website.com/env/local/lib/python2.7/site-packages/sqlalchemy/orm/query.py", line 2453, in _execute_and_instances
result = conn.execute(querycontext.statement, self._params)
File "/var/www/website.com/env/local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 729, in execute
return meth(self, multiparams, params)
File "/var/www/website.com/env/local/lib/python2.7/site-packages/sqlalchemy/sql/elements.py", line 322, in _execute_on_connection
return connection._execute_clauseelement(self, multiparams, params)
File "/var/www/website.com/env/local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 826, in _execute_clauseelement
compiled_sql, distilled_params
File "/var/www/website.com/env/local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 958, in _execute_context
context)
File "/var/www/website.com/env/local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1159, in _handle_dbapi_exception
exc_info
File "/var/www/website.com/env/local/lib/python2.7/site-packages/sqlalchemy/util/compat.py", line 199, in raise_from_cause
reraise(type(exception), exception, tb=exc_tb)
File "/var/www/website.com/env/local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 951, in _execute_context
context)
File "/var/www/website.com/env/local/lib/python2.7/site-packages/sqlalchemy/engine/default.py", line 436, in do_execute
cursor.execute(statement, parameters)
File "/var/www/website.com/env/local/lib/python2.7/site-packages/MySQLdb/cursors.py", line 205, in execute
self.errorhandler(self, exc, value)
File "/var/www/website.com/env/local/lib/python2.7/site-packages/MySQLdb/connections.py", line 36, in defaulterrorhandler
raise errorclass, errorvalue
OperationalError: (OperationalError) (2006, 'MySQL server has gone away') 'SELECT `Users`.id AS `Users_id`, `Users`.email AS `Users_email`, `Users`.password AS `Users_password`, `Users`.`emailConfirmed` AS `Users_emailConfirmed`, `Users`.`createdAtUtc` AS `Users_createdAtUtc`, `Users`.`updatedAtUtc` AS `Users_updatedAtUtc`, `Users`.`roleId` AS `Users_roleId` \nFROM `Users` \nWHERE `Users`.email = %s \n LIMIT %s' ('user#website.demo', 1)
2015-03-19 15:06:22,089 ERROR: Exception on /api/v1/account/login [POST] [in /var/www/website.com/env/local/lib/python2.7/site-packages/flask/app.py:1423]
Traceback (most recent call last):
File "/var/www/website.com/env/local/lib/python2.7/site-packages/flask/app.py", line 1817, in wsgi_app
response = self.full_dispatch_request()
File "/var/www/website.com/env/local/lib/python2.7/site-packages/flask/app.py", line 1477, in full_dispatch_request
rv = self.handle_user_exception(e)
File "/var/www/website.com/env/local/lib/python2.7/site-packages/flask_cors.py", line 272, in wrapped_function
return cors_after_request(app.make_response(f(*args, **kwargs)))
File "/var/www/website.com/env/local/lib/python2.7/site-packages/flask/app.py", line 1381, in handle_user_exception
reraise(exc_type, exc_value, tb)
File "/var/www/website.com/env/local/lib/python2.7/site-packages/flask/app.py", line 1475, in full_dispatch_request
rv = self.dispatch_request()
File "/var/www/website.com/env/local/lib/python2.7/site-packages/flask/app.py", line 1461, in dispatch_request
return self.view_functions[rule.endpoint](**req.view_args)
File "/var/www/website.com/env/local/lib/python2.7/site-packages/flask_classy.py", line 200, in proxy
response = view(**request.view_args)
File "./webportal/account/api.py", line 34, in login_user
user = auth.authenticate(username=email, password=password)
File "/var/www/website.com/server/webportal/auth.py", line 23, in authenticate
user = User.query.filter_by(email=username).first()
File "/var/www/website.com/env/local/lib/python2.7/site-packages/sqlalchemy/orm/query.py", line 2367, in first
ret = list(self[0:1])
File "/var/www/website.com/env/local/lib/python2.7/site-packages/sqlalchemy/orm/query.py", line 2228, in __getitem__
return list(res)
File "/var/www/website.com/env/local/lib/python2.7/site-packages/sqlalchemy/orm/query.py", line 2438, in __iter__
return self._execute_and_instances(context)
File "/var/www/website.com/env/local/lib/python2.7/site-packages/sqlalchemy/orm/query.py", line 2453, in _execute_and_instances
result = conn.execute(querycontext.statement, self._params)
File "/var/www/website.com/env/local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 729, in execute
return meth(self, multiparams, params)
File "/var/www/website.com/env/local/lib/python2.7/site-packages/sqlalchemy/sql/elements.py", line 322, in _execute_on_connection
return connection._execute_clauseelement(self, multiparams, params)
File "/var/www/website.com/env/local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 826, in _execute_clauseelement
compiled_sql, distilled_params
File "/var/www/website.com/env/local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 893, in _execute_context
None, None)
File "/var/www/website.com/env/local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1159, in _handle_dbapi_exception
exc_info
File "/var/www/website.com/env/local/lib/python2.7/site-packages/sqlalchemy/util/compat.py", line 199, in raise_from_cause
reraise(type(exception), exception, tb=exc_tb)
File "/var/www/website.com/env/local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 887, in _execute_context
conn = self._revalidate_connection()
File "/var/www/website.com/env/local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 242, in _revalidate_connection
"Can't reconnect until invalid "
StatementError: Can't reconnect until invalid transaction is rolled back (original cause: InvalidRequestError: Can't reconnect until invalid transaction is rolled back) u'SELECT `Users`.id AS `Users_id`, `Users`.email AS `Users_email`, `Users`.password AS `Users_password`, `Users`.`emailConfirmed` AS `Users_emailConfirmed`, `Users`.`createdAtUtc` AS `Users_createdAtUtc`, `Users`.`updatedAtUtc` AS `Users_updatedAtUtc`, `Users`.`roleId` AS `Users_roleId` \nFROM `Users` \nWHERE `Users`.email = %s \n LIMIT %s' [immutabledict({})]
According to https://dev.mysql.com/doc/refman/5.0/en/gone-away.html, MySQL closes the connection after 8 hours... Once that happens, the app gets in a bit of a state.

Resources