my ~/.aws/credentials looks like
[default]
aws_access_key_id = XYZ
aws_secret_access_key = ABC
[testing]
source_profile = default
role_arn = arn:aws:iam::54:role/ad
I add my remote like
dvc remote add --local -v myremote s3://bib-ds-models-testing/data/dvc-test
I have made my .dvc/config.local to look like
[‘remote “myremote”’]
url = s3://bib-ds-models-testing/data/dvc-test
access_key_id = XYZ
secret_access_key = ABC/h2hOsRcCIFqwYWV7eZaUq3gNmS
profile=‘testing’
credentialpath = /Users/nyt21/.aws/credentials
but still after running dvc push -r myremote I get
ERROR: unexpected error - Forbidden: An error occurred (403) when calling the HeadObject operation: Forbidden
** Update
here is the output of dvc push -v
2021-07-25 22:40:38,887 DEBUG: Check for update is enabled.
2021-07-25 22:40:39,022 DEBUG: Preparing to upload data to 's3://bib-ds-models-testing/data/dvc-test'
2021-07-25 22:40:39,022 DEBUG: Preparing to collect status from s3://bib-ds-models-testing/data/dvc-test
2021-07-25 22:40:39,022 DEBUG: Collecting information from local cache...
2021-07-25 22:40:39,022 DEBUG: Collecting information from remote cache...
2021-07-25 22:40:39,022 DEBUG: Matched '0' indexed hashes
2021-07-25 22:40:39,022 DEBUG: Querying 1 hashes via object_exists
2021-07-25 22:40:39,644 ERROR: unexpected error - Forbidden: An error occurred (403) when calling the HeadObject operation: Forbidden
------------------------------------------------------------
Traceback (most recent call last):
File "/Users/nyt21/opt/miniconda3/envs/dvc/lib/python3.8/site-packages/s3fs/core.py", line 246, in _call_s3
out = await method(**additional_kwargs)
File "/Users/nyt21/opt/miniconda3/envs/dvc/lib/python3.8/site-packages/aiobotocore/client.py", line 154, in _make_api_call
raise error_class(parsed_response, operation_name)
botocore.exceptions.ClientError: An error occurred (AccessDenied) when calling the ListObjectsV2 operation: Access Denied
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/Users/nyt21/opt/miniconda3/envs/dvc/lib/python3.8/site-packages/s3fs/core.py", line 1057, in _info
out = await self._simple_info(path)
File "/Users/nyt21/opt/miniconda3/envs/dvc/lib/python3.8/site-packages/s3fs/core.py", line 970, in _simple_info
out = await self._call_s3(
File "/Users/nyt21/opt/miniconda3/envs/dvc/lib/python3.8/site-packages/s3fs/core.py", line 265, in _call_s3
raise translate_boto_error(err)
PermissionError: Access Denied
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/Users/nyt21/opt/miniconda3/envs/dvc/lib/python3.8/site-packages/s3fs/core.py", line 246, in _call_s3
out = await method(**additional_kwargs)
File "/Users/nyt21/opt/miniconda3/envs/dvc/lib/python3.8/site-packages/aiobotocore/client.py", line 154, in _make_api_call
raise error_class(parsed_response, operation_name)
botocore.exceptions.ClientError: An error occurred (403) when calling the HeadObject operation: Forbidden
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/Users/nyt21/opt/miniconda3/envs/dvc/lib/python3.8/site-packages/dvc/main.py", line 55, in main
ret = cmd.do_run()
File "/Users/nyt21/opt/miniconda3/envs/dvc/lib/python3.8/site-packages/dvc/command/base.py", line 50, in do_run
return self.run()
File "/Users/nyt21/opt/miniconda3/envs/dvc/lib/python3.8/site-packages/dvc/command/data_sync.py", line 57, in run
processed_files_count = self.repo.push(
File "/Users/nyt21/opt/miniconda3/envs/dvc/lib/python3.8/site-packages/dvc/repo/__init__.py", line 51, in wrapper
return f(repo, *args, **kwargs)
File "/Users/nyt21/opt/miniconda3/envs/dvc/lib/python3.8/site-packages/dvc/repo/push.py", line 44, in push
pushed += self.cloud.push(objs, jobs, remote=remote)
File "/Users/nyt21/opt/miniconda3/envs/dvc/lib/python3.8/site-packages/dvc/data_cloud.py", line 79, in push
return remote_obj.push(
File "/Users/nyt21/opt/miniconda3/envs/dvc/lib/python3.8/site-packages/dvc/remote/base.py", line 57, in wrapper
return f(obj, *args, **kwargs)
File "/Users/nyt21/opt/miniconda3/envs/dvc/lib/python3.8/site-packages/dvc/remote/base.py", line 494, in push
ret = self._process(
File "/Users/nyt21/opt/miniconda3/envs/dvc/lib/python3.8/site-packages/dvc/remote/base.py", line 351, in _process
dir_status, file_status, dir_contents = self._status(
File "/Users/nyt21/opt/miniconda3/envs/dvc/lib/python3.8/site-packages/dvc/remote/base.py", line 195, in _status
self.hashes_exist(
File "/Users/nyt21/opt/miniconda3/envs/dvc/lib/python3.8/site-packages/dvc/remote/base.py", line 145, in hashes_exist
return indexed_hashes + self.odb.hashes_exist(list(hashes), **kwargs)
File "/Users/nyt21/opt/miniconda3/envs/dvc/lib/python3.8/site-packages/dvc/objects/db/base.py", line 438, in hashes_exist
remote_hashes = self.list_hashes_exists(hashes, jobs, name)
File "/Users/nyt21/opt/miniconda3/envs/dvc/lib/python3.8/site-packages/dvc/objects/db/base.py", line 389, in list_hashes_exists
ret = list(itertools.compress(hashes, in_remote))
File "/Users/nyt21/opt/miniconda3/envs/dvc/lib/python3.8/concurrent/futures/_base.py", line 619, in result_iterator
yield fs.pop().result()
File "/Users/nyt21/opt/miniconda3/envs/dvc/lib/python3.8/concurrent/futures/_base.py", line 444, in result
return self.__get_result()
File "/Users/nyt21/opt/miniconda3/envs/dvc/lib/python3.8/concurrent/futures/_base.py", line 389, in __get_result
raise self._exception
File "/Users/nyt21/opt/miniconda3/envs/dvc/lib/python3.8/concurrent/futures/thread.py", line 57, in run
result = self.fn(*self.args, **self.kwargs)
File "/Users/nyt21/opt/miniconda3/envs/dvc/lib/python3.8/site-packages/dvc/objects/db/base.py", line 380, in exists_with_progress
ret = self.fs.exists(path_info)
File "/Users/nyt21/opt/miniconda3/envs/dvc/lib/python3.8/site-packages/dvc/fs/fsspec_wrapper.py", line 92, in exists
return self.fs.exists(self._with_bucket(path_info))
File "/Users/nyt21/opt/miniconda3/envs/dvc/lib/python3.8/site-packages/fsspec/asyn.py", line 87, in wrapper
return sync(self.loop, func, *args, **kwargs)
File "/Users/nyt21/opt/miniconda3/envs/dvc/lib/python3.8/site-packages/fsspec/asyn.py", line 68, in sync
raise result[0]
File "/Users/nyt21/opt/miniconda3/envs/dvc/lib/python3.8/site-packages/fsspec/asyn.py", line 24, in _runner
result[0] = await coro
File "/Users/nyt21/opt/miniconda3/envs/dvc/lib/python3.8/site-packages/s3fs/core.py", line 802, in _exists
await self._info(path, bucket, key, version_id=version_id)
File "/Users/nyt21/opt/miniconda3/envs/dvc/lib/python3.8/site-packages/s3fs/core.py", line 1061, in _info
out = await self._version_aware_info(path, version_id)
File "/Users/nyt21/opt/miniconda3/envs/dvc/lib/python3.8/site-packages/s3fs/core.py", line 1004, in _version_aware_info
out = await self._call_s3(
File "/Users/nyt21/opt/miniconda3/envs/dvc/lib/python3.8/site-packages/s3fs/core.py", line 265, in _call_s3
raise translate_boto_error(err)
PermissionError: Forbidden
------------------------------------------------------------
2021-07-25 22:40:39,712 DEBUG: Version info for developers:
DVC version: 2.5.4 (pip)
---------------------------------
Platform: Python 3.8.10 on macOS-10.16-x86_64-i386-64bit
Supports:
http (requests = 2.26.0),
https (requests = 2.26.0),
s3 (s3fs = 2021.6.1, boto3 = 1.18.6)
Cache types: reflink, hardlink, symlink
Cache directory: apfs on /dev/disk3s1s1
Caches: local
Remotes: s3
Workspace directory: apfs on /dev/disk3s1s1
Repo: dvc, git
Having any troubles? Hit us up at https://dvc.org/support, we are always happy to help!
2021-07-25 22:40:39,713 DEBUG: Analytics is enabled.
2021-07-25 22:40:39,765 DEBUG: Trying to spawn '['daemon', '-q', 'analytics', '/var/folders/4x/xhm22wt16gl6m9nvkl9gllkc0000gn/T/tmpo86jdns5']'
2021-07-25 22:40:39,769 DEBUG: Spawned '['daemon', '-q', 'analytics', '/var/folders/4x/xhm22wt16gl6m9nvkl9gllkc0000gn/T/tmpo86jdns5']'
I can upload through python
import boto3
import os
import pickle
bucket_name = 'bib-ds-models-testing'
os.environ["AWS_PROFILE"] = "testing"
session = boto3.Session()
s3_client = boto3.client('s3')
s3_client.upload_file('/Users/nyt21/Devel/DVC/test/data/iris.csv',
'bib-ds-models-testing',
'data/dvc-test/my_iris.csv')
I don't use aws CLI but the following also gives an access deny !
aws s3 ls s3://bib-ds-models-testing/data/dvc-test
An error occurred (AccessDenied) when calling the ListObjectsV2
operation: Access Denied
but it works if I add --profile=testing
aws s3 ls s3://bib-ds-models-testing/data/dvc-test --profile=testing
PRE dvc-test/
just you know environment variable AWS_PROFILE is already set to 'testing'
UPDATE
I have tried both AWS_PROFILE='testing' and AWS_PROFILE=testing, neither of them worked.
Related
We deployed Apache Airflow 2.3.3 to Azure.
Webserver - Web App
Scheduler - ACI
Celery Worker - ACI
We were seeing errors on the Celery ACI console related to Postgres and Redis connection timeouts as below
[2022-09-22 18:55:50,650: WARNING/ForkPoolWorker-15] Failed operation _store_result. Retrying 2 more times.
Traceback (most recent call last):
File "/home/airflow/.local/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 1803, in _execute_context
cursor, statement, parameters, context
File "/home/airflow/.local/lib/python3.7/site-packages/sqlalchemy/engine/default.py", line 719, in do_execute
cursor.execute(statement, parameters)
psycopg2.DatabaseError: could not receive data from server: Connection timed out
SSL SYSCALL error: Connection timed out
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/home/airflow/.local/lib/python3.7/site-packages/celery/backends/database/__init__.py", line 47, in _inner
return fun(*args, **kwargs)
File "/home/airflow/.local/lib/python3.7/site-packages/celery/backends/database/__init__.py", line 117, in _store_result
task = list(session.query(self.task_cls).filter(self.task_cls.task_id == task_id))
File "/home/airflow/.local/lib/python3.7/site-packages/sqlalchemy/orm/query.py", line 2887, in __iter__
return self._iter().__iter__()
File "/home/airflow/.local/lib/python3.7/site-packages/sqlalchemy/orm/query.py", line 2897, in _iter
execution_options={"_sa_orm_load_options": self.load_options},
File "/home/airflow/.local/lib/python3.7/site-packages/sqlalchemy/orm/session.py", line 1689, in execute
result = conn._execute_20(statement, params or {}, execution_options)
File "/home/airflow/.local/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 1614, in _execute_20
return meth(self, args_10style, kwargs_10style, execution_options)
File "/home/airflow/.local/lib/python3.7/site-packages/sqlalchemy/sql/elements.py", line 326, in _execute_on_connection
self, multiparams, params, execution_options
File "/home/airflow/.local/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 1491, in _execute_clauseelement
cache_hit=cache_hit,
File "/home/airflow/.local/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 1846, in _execute_context
e, statement, parameters, cursor, context
File "/home/airflow/.local/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 2027, in _handle_dbapi_exception
sqlalchemy_exception, with_traceback=exc_info[2], from_=e
File "/home/airflow/.local/lib/python3.7/site-packages/sqlalchemy/util/compat.py", line 207, in raise_
raise exception
File "/home/airflow/.local/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 1803, in _execute_context
cursor, statement, parameters, context
File "/home/airflow/.local/lib/python3.7/site-packages/sqlalchemy/engine/default.py", line 719, in do_execute
cursor.execute(statement, parameters)
sqlalchemy.exc.DatabaseError: (psycopg2.DatabaseError) could not receive data from server: Connection timed out
SSL SYSCALL error: Connection timed out
[SQL: SELECT celery_taskmeta.id AS celery_taskmeta_id, celery_taskmeta.task_id AS celery_taskmeta_task_id, celery_taskmeta.status AS celery_taskmeta_status, celery_taskmeta.result AS celery_taskmeta_result, celery_taskmeta.date_done AS celery_taskmeta_date_done, celery_taskmeta.traceback AS celery_taskmeta_traceback
FROM celery_taskmeta
WHERE celery_taskmeta.task_id = %(task_id_1)s]
[parameters: {'task_id_1': 'c5f9f53c-8afe-4d67-8d3b-d7ad84875de1'}]
(Background on this error at: https://sqlalche.me/e/14/4xp6)
[2022-09-22 18:55:50,929: INFO/ForkPoolWorker-15] [c5f9f53c-8afe-4d67-8d3b-d7ad84875de1] Executing command in Celery: ['airflow', 'tasks', 'run', 'CS_ALERTING', 'CheckRunningTasks', 'scheduled__2022-09-22T18:00:00+00:00', '--local', '--subdir', 'DAGS_FOLDER/CS_ALERTING.py']
[2022-09-22 18:55:51,241: INFO/ForkPoolWorker-15] Filling up the DagBag from /opt/airflow/platform_pam/dags/CS_ALERTING.py
[2022-09-22 18:55:53,467: INFO/ForkPoolWorker-15] Running <TaskInstance: CS_ALERTING.CheckRunningTasks scheduled__2022-09-22T18:00:00+00:00 [queued]> on host localhost
[2022-09-22 18:55:58,304: INFO/ForkPoolWorker-15] Task airflow.executors.celery_executor.execute_command[c5f9f53c-8afe-4d67-8d3b-d7ad84875de1] succeeded in 960.1964174450004s: None
[2022-09-22 19:29:25,931: WARNING/MainProcess] consumer: Connection to broker lost. Trying to re-establish the connection...
Traceback (most recent call last):
File "/home/airflow/.local/lib/python3.7/site-packages/redis/connection.py", line 706, in send_packed_command
sendall(self._sock, item)
File "/home/airflow/.local/lib/python3.7/site-packages/redis/_compat.py", line 9, in sendall
return sock.sendall(*args, **kwargs)
File "/usr/local/lib/python3.7/ssl.py", line 1034, in sendall
v = self.send(byte_view[count:])
File "/usr/local/lib/python3.7/ssl.py", line 1003, in send
return self._sslobj.write(data)
TimeoutError: [Errno 110] Connection timed out
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/airflow/.local/lib/python3.7/site-packages/celery/worker/consumer/consumer.py", line 332, in start
blueprint.start(self)
File "/home/airflow/.local/lib/python3.7/site-packages/celery/bootsteps.py", line 116, in start
step.start(parent)
File "/home/airflow/.local/lib/python3.7/site-packages/celery/worker/consumer/consumer.py", line 628, in start
c.loop(*c.loop_args())
File "/home/airflow/.local/lib/python3.7/site-packages/celery/worker/loops.py", line 97, in asynloop
next(loop)
File "/home/airflow/.local/lib/python3.7/site-packages/kombu/asynchronous/hub.py", line 301, in create_loop
poll_timeout = fire_timers(propagate=propagate) if scheduled else 1
File "/home/airflow/.local/lib/python3.7/site-packages/kombu/asynchronous/hub.py", line 143, in fire_timers
entry()
File "/home/airflow/.local/lib/python3.7/site-packages/kombu/asynchronous/timer.py", line 64, in __call__
return self.fun(*self.args, **self.kwargs)
File "/home/airflow/.local/lib/python3.7/site-packages/kombu/asynchronous/timer.py", line 126, in _reschedules
return fun(*args, **kwargs)
File "/home/airflow/.local/lib/python3.7/site-packages/kombu/transport/redis.py", line 557, in maybe_check_subclient_health
client.check_health()
File "/home/airflow/.local/lib/python3.7/site-packages/redis/client.py", line 3522, in check_health
check_health=False)
File "/home/airflow/.local/lib/python3.7/site-packages/redis/connection.py", line 726, in send_command
check_health=kwargs.get('check_health', True))
File "/home/airflow/.local/lib/python3.7/site-packages/redis/connection.py", line 718, in send_packed_command
(errno, errmsg))
redis.exceptions.ConnectionError: Error 110 while writing to socket. Connection timed out.
[2022-09-22 19:29:26,023: WARNING/MainProcess] /home/airflow/.local/lib/python3.7/site-packages/celery/worker/consumer/consumer.py:367: CPendingDeprecationWarning:
I referred the Airflow's documentation and found setting up database
We are modifying Airflow's docker image and adding a python file,
airflow.www.db_utils.db_config (This file is installed to site_packages) and defined the dictionary
keepalive_kwargs = {
"keepalives": 1,
"keepalives_idle": 30,
"keepalives_interval": 5,
"keepalives_count": 5,
}
Finally, we are setting
ENV AIRFLOW__DATABASE__SQL_ALCHEMY_CONNECT_ARGS="airflow.www.db_utils.db_config.keepalive_kwargs"
Unfortunately, the error stills persist. It will be great if someone helps me to resolve this issue.
Our webserver is running and working fine. Although in the logs we are seeing these errors periodically . Can someone provide pointers around these
[2022-04-05 02:53:58 -0400] [129502] [ERROR] Socket error processing request.
Traceback (most recent call last):
File "/airflow/airflowvirt/lib64/python3.6/site-packages/gunicorn/workers/sync.py", line 135, in handle
req = next(parser)
File "/airflow/airflowvirt/lib64/python3.6/site-packages/gunicorn/http/parser.py", line 42, in __next__
self.mesg = self.mesg_class(self.cfg, self.unreader, self.source_addr, self.req_count)
File "/airflow/airflowvirt/lib64/python3.6/site-packages/gunicorn/http/message.py", line 180, in __init__
super().__init__(cfg, unreader, peer_addr)
File "/airflow/airflowvirt/lib64/python3.6/site-packages/gunicorn/http/message.py", line 54, in __init__
unused = self.parse(self.unreader)
File "/airflow/airflowvirt/lib64/python3.6/site-packages/gunicorn/http/message.py", line 192, in parse
self.get_data(unreader, buf, stop=True)
File "/airflow/airflowvirt/lib64/python3.6/site-packages/gunicorn/http/message.py", line 183, in get_data
data = unreader.read()
File "/airflow/airflowvirt/lib64/python3.6/site-packages/gunicorn/http/unreader.py", line 37, in read
d = self.chunk()
File "/airflow/airflowvirt/lib64/python3.6/site-packages/gunicorn/http/unreader.py", line 64, in chunk
return self.sock.recv(self.mxchunk)
File "/usr/lib64/python3.6/ssl.py", line 956, in recv
return self.read(buflen)
File "/usr/lib64/python3.6/ssl.py", line 833, in read
return self._sslobj.read(len, buffer)
File "/usr/lib64/python3.6/ssl.py", line 592, in read
v = self._sslobj.read(len)
OSError: [Errno 0] Error
[2022-04-05 02:53:58 -0400] [129500] [ERROR] Socket error processing request.
I have installed the spacy 3.1.2 and trying to install en_core_web_sm on jupyter notebook in Jupyterlab by using
!python3 -m spacy download en_core_web_sm
but it is showing the following error
2021-09-01 07:00:12.028366: W tensorflow/stream_executor/platform/default/dso_loader.cc:55] Could not load dynamic library 'libnvinfer.so.6'; dlerror: libnvinfer.so.6: cannot open shared object file: No such file or directory
2021-09-01 07:00:12.028469: W tensorflow/stream_executor/platform/default/dso_loader.cc:55] Could not load dynamic library 'libnvinfer_plugin.so.6'; dlerror: libnvinfer_plugin.so.6: cannot open shared object file: No such file or directory
2021-09-01 07:00:12.028486: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:30] Cannot dlopen some TensorRT libraries. If you would like to use Nvidia GPU with TensorRT, please make sure the missing libraries mentioned above are installed properly.
Traceback (most recent call last):
File "/opt/conda/lib/python3.7/site-packages/urllib3/contrib/pyopenssl.py", line 488, in wrap_socket
cnx.do_handshake()
File "/opt/conda/lib/python3.7/site-packages/OpenSSL/SSL.py", line 1934, in do_handshake
self._raise_ssl_error(self._ssl, result)
File "/opt/conda/lib/python3.7/site-packages/OpenSSL/SSL.py", line 1671, in _raise_ssl_error
_raise_current_error()
File "/opt/conda/lib/python3.7/site-packages/OpenSSL/_util.py", line 54, in exception_from_error_queue
raise exception_type(errors)
OpenSSL.SSL.Error: [('SSL routines', 'tls_process_server_certificate', 'certificate verify failed')]
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/opt/conda/lib/python3.7/site-packages/urllib3/connectionpool.py", line 667, in urlopen
self._prepare_proxy(conn)
File "/opt/conda/lib/python3.7/site-packages/urllib3/connectionpool.py", line 932, in _prepare_proxy
conn.connect()
File "/opt/conda/lib/python3.7/site-packages/urllib3/connection.py", line 371, in connect
ssl_context=context,
File "/opt/conda/lib/python3.7/site-packages/urllib3/util/ssl_.py", line 384, in ssl_wrap_socket
return context.wrap_socket(sock, server_hostname=server_hostname)
File "/opt/conda/lib/python3.7/site-packages/urllib3/contrib/pyopenssl.py", line 494, in wrap_socket
raise ssl.SSLError("bad handshake: %r" % e)
ssl.SSLError: ("bad handshake: Error([('SSL routines', 'tls_process_server_certificate', 'certificate verify failed')])",)
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/opt/conda/lib/python3.7/site-packages/requests/adapters.py", line 449, in send
timeout=timeout
File "/opt/conda/lib/python3.7/site-packages/urllib3/connectionpool.py", line 727, in urlopen
method, url, error=e, _pool=self, _stacktrace=sys.exc_info()[2]
File "/opt/conda/lib/python3.7/site-packages/urllib3/util/retry.py", line 439, in increment
raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='raw.githubusercontent.com', port=443): Max retries exceeded with url: /explosion/spacy-models/master/compatibility.json (Caused by SSLError(SSLError("bad handshake: Error([('SSL routines', 'tls_process_server_certificate', 'certificate verify failed')])")))
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/opt/conda/lib/python3.7/runpy.py", line 193, in _run_module_as_main
"__main__", mod_spec)
File "/opt/conda/lib/python3.7/runpy.py", line 85, in _run_code
exec(code, run_globals)
File "/home/jovyan/.local/lib/python3.7/site-packages/spacy/__main__.py", line 4, in <module>
setup_cli()
File "/home/jovyan/.local/lib/python3.7/site-packages/spacy/cli/_util.py", line 69, in setup_cli
command(prog_name=COMMAND)
File "/opt/conda/lib/python3.7/site-packages/click/core.py", line 829, in __call__
return self.main(*args, **kwargs)
File "/opt/conda/lib/python3.7/site-packages/click/core.py", line 782, in main
rv = self.invoke(ctx)
File "/opt/conda/lib/python3.7/site-packages/click/core.py", line 1259, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/opt/conda/lib/python3.7/site-packages/click/core.py", line 1066, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "/opt/conda/lib/python3.7/site-packages/click/core.py", line 610, in invoke
return callback(*args, **kwargs)
File "/home/jovyan/.local/lib/python3.7/site-packages/typer/main.py", line 497, in wrapper
return callback(**use_params) # type: ignore
File "/home/jovyan/.local/lib/python3.7/site-packages/spacy/cli/download.py", line 35, in download_cli
download(model, direct, sdist, *ctx.args)
File "/home/jovyan/.local/lib/python3.7/site-packages/spacy/cli/download.py", line 67, in download
compatibility = get_compatibility()
File "/home/jovyan/.local/lib/python3.7/site-packages/spacy/cli/download.py", line 78, in get_compatibility
r = requests.get(about.__compatibility__)
File "/opt/conda/lib/python3.7/site-packages/requests/api.py", line 76, in get
return request('get', url, params=params, **kwargs)
File "/opt/conda/lib/python3.7/site-packages/requests/api.py", line 61, in request
return session.request(method=method, url=url, **kwargs)
File "/opt/conda/lib/python3.7/site-packages/requests/sessions.py", line 530, in request
resp = self.send(prep, **send_kwargs)
File "/opt/conda/lib/python3.7/site-packages/requests/sessions.py", line 643, in send
r = adapter.send(request, **kwargs)
File "/opt/conda/lib/python3.7/site-packages/requests/adapters.py", line 514, in send
raise SSLError(e, request=request)
requests.exceptions.SSLError: HTTPSConnectionPool(host='raw.githubusercontent.com', port=443): Max retries exceeded with url: /explosion/spacy-models/master/compatibility.json (Caused by SSLError(SSLError("bad handshake: Error([('SSL routines', 'tls_process_server_certificate', 'certificate verify failed')])")))
Due to this, I can't use
import spacy
nlp = spacy.load('en_core_web_sm')
Because it is showing error as:
OSError: [E050] Can't find model 'en_core_web_sm'. It doesn't seem to be a Python package or a valid path to a data directory.
I have searched various post and articles related to it but nothing is working for me.
Can anyone tell me how to resolve this error ? It would be very helpful for me.
I am trying to use the responses module to mock an HTTP server and I have the feeling it is broken when calls requests are multi-threaded.
Consider the following:
import json
import responses
import requests
import threading
try:
import urlparse
except ModuleNotFoundError:
import urllib.parse as urlparse
server_url = 'http://server'
headers_json = {'content-type': 'application/json'}
def init():
def endpoint(request):
id = urlparse.parse_qs(urlparse.urlparse(request.path_url).query)["id"][0]
data = {"foo": int(id)}
return 200, headers_json, json.dumps(data)
responses.add_callback(
responses.GET, server_url + '/',
callback=endpoint,
)
#responses.activate
def test():
init()
def responses_routine():
resp = requests.get(
server_url + '/?id=456',
headers=headers_json,
)
# {"foo": "456"}
print(resp.json()["foo"])
def print_routine():
print("456")
# This will print "456"
#responses_routine()
# This will print "456" after 2 seconds
#threading.Timer(2, print_routine).start()
# This will fail
threading.Timer(2, responses_routine).start()
test()
It fails with a stack that clearly indicates that no call to responses was made. The requests module did try to make an HTTP code to a remote server.
Exception in thread Thread-1:
Traceback (most recent call last):
File "/opt/python/python3.6.6/lib/python3.6/site-packages/urllib3/connection.py", line 159, in _new_conn
(self._dns_host, self.port), self.timeout, **extra_kw)
File "/opt/python/python3.6.6/lib/python3.6/site-packages/urllib3/util/connection.py", line 57, in create_connection
for res in socket.getaddrinfo(host, port, family, socket.SOCK_STREAM):
File "/opt/python/python3.6.6/lib/python3.6/socket.py", line 745, in getaddrinfo
for res in _socket.getaddrinfo(host, port, family, type, proto, flags):
socket.gaierror: [Errno -2] Name or service not known
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/opt/python/python3.6.6/lib/python3.6/site-packages/urllib3/connectionpool.py", line 600, in urlopen
chunked=chunked)
File "/opt/python/python3.6.6/lib/python3.6/site-packages/urllib3/connectionpool.py", line 354, in _make_request
conn.request(method, url, **httplib_request_kw)
File "/opt/python/python3.6.6/lib/python3.6/http/client.py", line 1239, in request
self._send_request(method, url, body, headers, encode_chunked)
File "/opt/python/python3.6.6/lib/python3.6/http/client.py", line 1285, in _send_request
self.endheaders(body, encode_chunked=encode_chunked)
File "/opt/python/python3.6.6/lib/python3.6/http/client.py", line 1234, in endheaders
self._send_output(message_body, encode_chunked=encode_chunked)
File "/opt/python/python3.6.6/lib/python3.6/http/client.py", line 1026, in _send_output
self.send(msg)
File "/opt/python/python3.6.6/lib/python3.6/http/client.py", line 964, in send
self.connect()
File "/opt/python/python3.6.6/lib/python3.6/site-packages/urllib3/connection.py", line 181, in connect
conn = self._new_conn()
File "/opt/python/python3.6.6/lib/python3.6/site-packages/urllib3/connection.py", line 168, in _new_conn
self, "Failed to establish a new connection: %s" % e)
urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPConnection object at 0x2aaab0930438>: Failed to establish a new connection: [Errno -2] Name or service not known
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/opt/python/python3.6.6/lib/python3.6/site-packages/requests/adapters.py", line 449, in send
timeout=timeout
File "/opt/python/python3.6.6/lib/python3.6/site-packages/urllib3/connectionpool.py", line 638, in urlopen
_stacktrace=sys.exc_info()[2])
File "/opt/python/python3.6.6/lib/python3.6/site-packages/urllib3/util/retry.py", line 398, in increment
raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPConnectionPool(host='server', port=80): Max retries exceeded with url: /?id=456 (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x2aaab0930438>: Failed to establish a new connection: [Errno -2] Name or service not known',))
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/opt/python/python3.6.6/lib/python3.6/threading.py", line 916, in _bootstrap_inner
self.run()
File "/opt/python/python3.6.6/lib/python3.6/threading.py", line 1182, in run
self.function(*self.args, **self.kwargs)
File "tmp.py", line 35, in responses_routine
headers=headers_json,
File "/opt/python/python3.6.6/lib/python3.6/site-packages/requests/api.py", line 116, in post
return request('post', url, data=data, json=json, **kwargs)
File "/opt/python/python3.6.6/lib/python3.6/site-packages/requests/api.py", line 60, in request
return session.request(method=method, url=url, **kwargs)
File "/opt/python/python3.6.6/lib/python3.6/site-packages/requests/sessions.py", line 533, in request
resp = self.send(prep, **send_kwargs)
File "/opt/python/python3.6.6/lib/python3.6/site-packages/requests/sessions.py", line 646, in send
r = adapter.send(request, **kwargs)
File "/opt/python/python3.6.6/lib/python3.6/site-packages/requests/adapters.py", line 516, in send
raise ConnectionError(e, request=request)
requests.exceptions.ConnectionError: HTTPConnectionPool(host='server', port=80): Max retries exceeded with url: /?id=456 (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x2aaab0930438>: Failed to establish a new connection: [Errno -2] Name or service not known',))
It fails on Redhat 6.6. I am using Python 3.6.6 (it also fails in 2.7 with a similar stack) with requests 2.22 and responses 0.10.6.
It's like when the main thread exited, all "handles" to responses were lost...
Any clues ?
Actually what's created within the context manager responses.activate should not escape it.
In the code above, the underlying thread of the Timer object would be joined outside the scope, hence potentially run outside the context manager scope, which yield the exception.
I am trying to install salt stack while running the command
python C:\Python27\Scripts\salt-minion -c C:\salt\minion -l debug i am getting error message like Master hostname: salt not found. Retrying in 30 seconds
I am attaching my error report please find it
C:\salt>python C:\Python27\Scripts\salt-minion -c C:\salt\minion -l debug
WARNING: 'C:\\salt\\minion' directory does not exist.
[DEBUG ] Missing configuration file: C:\salt\minion\minion
[DEBUG ] Guessing ID. The id can be explicitly in set c:\salt\conf\minion
[INFO ] Found minion id from get_fqhostname(): HoroppaLabs
[ERROR ] Could not cache minion ID: [Errno 2] No such file or directory: 'c:\\
salt\\conf\\minion_id'
[DEBUG ] Configuration file path: c:\salt\conf\minion
[WARNING ] Failed to open log file, do you have permission to write to c:\salt\v
ar\log\salt\minion?
[INFO ] Setting up the Salt Minion "HoroppaLabs"
[DEBUG ] Created pidfile: c:\salt\var\run\salt-minion.pid
[WARNING ] You have a version of ZMQ less than ZMQ 3.2! There are known connecti
on keep-alive issues with ZMQ < 3.2 which may result in loss of contact with min
ions. Please upgrade your ZMQ!
[DEBUG ] Missing configuration file: c:\salt\conf\minion
[ERROR ] Unable to import Python wmi module, some core grains will be missing
Traceback (most recent call last):
File "C:\Python27\lib\site-packages\salt\grains\core.py", line 50, in <module>
import wmi
ImportError: No module named wmi
[CRITICAL] Failed to load grains defined in grain file core.hwaddr_interfaces in
function <function hwaddr_interfaces at 0x00000000037AA048>, error:
Traceback (most recent call last):
File "C:\Python27\lib\site-packages\salt\loader.py", line 1127, in gen_grains
ret = fun()
File "C:\Python27\lib\site-packages\salt\grains\core.py", line 1180, in hwaddr
_interfaces
ifaces = salt.utils.network.interfaces()
File "C:\Python27\lib\site-packages\salt\utils\network.py", line 449, in inter
faces
return win_interfaces()
File "C:\Python27\lib\site-packages\salt\utils\network.py", line 402, in win_i
nterfaces
with salt.utils.winapi.Com():
AttributeError: 'module' object has no attribute 'winapi'
[CRITICAL] Failed to load grains defined in grain file core.ip6 in function <fun
ction ip6 at 0x00000000037A5EB8>, error:
Traceback (most recent call last):
File "C:\Python27\lib\site-packages\salt\loader.py", line 1127, in gen_grains
ret = fun()
File "C:\Python27\lib\site-packages\salt\grains\core.py", line 1129, in ip6
return {'ipv6': salt.utils.network.ip_addrs6(include_loopback=True)}
File "C:\Python27\lib\site-packages\salt\utils\network.py", line 577, in ip_ad
drs6
ifaces = interfaces()
File "C:\Python27\lib\site-packages\salt\utils\network.py", line 449, in inter
faces
return win_interfaces()
File "C:\Python27\lib\site-packages\salt\utils\network.py", line 402, in win_i
nterfaces
with salt.utils.winapi.Com():
AttributeError: 'module' object has no attribute 'winapi'
[CRITICAL] Failed to load grains defined in grain file core.ip4 in function <fun
ction ip4 at 0x00000000037A5DD8>, error:
Traceback (most recent call last):
File "C:\Python27\lib\site-packages\salt\loader.py", line 1127, in gen_grains
ret = fun()
File "C:\Python27\lib\site-packages\salt\grains\core.py", line 1102, in ip4
return {'ipv4': salt.utils.network.ip_addrs(include_loopback=True)}
File "C:\Python27\lib\site-packages\salt\utils\network.py", line 546, in ip_ad
drs
ifaces = interfaces()
File "C:\Python27\lib\site-packages\salt\utils\network.py", line 449, in inter
faces
return win_interfaces()
File "C:\Python27\lib\site-packages\salt\utils\network.py", line 402, in win_i
nterfaces
with salt.utils.winapi.Com():
AttributeError: 'module' object has no attribute 'winapi'
[CRITICAL] Failed to load grains defined in grain file core.ip_interfaces in fun
ction <function ip_interfaces at 0x00000000037A5F98>, error:
Traceback (most recent call last):
File "C:\Python27\lib\site-packages\salt\loader.py", line 1127, in gen_grains
ret = fun()
File "C:\Python27\lib\site-packages\salt\grains\core.py", line 1159, in ip_int
erfaces
ifaces = salt.utils.network.interfaces()
File "C:\Python27\lib\site-packages\salt\utils\network.py", line 449, in inter
faces
return win_interfaces()
File "C:\Python27\lib\site-packages\salt\utils\network.py", line 402, in win_i
nterfaces
with salt.utils.winapi.Com():
AttributeError: 'module' object has no attribute 'winapi'
[ERROR ] This master address: 'salt' was previously resolvable but now fails t
o resolve! The previously resolved ip addr will continue to be used
[WARNING ] Master hostname: salt not found. Retrying in 30 seconds
Did you use the Salt Windows installer found here? http://docs.saltstack.com/en/latest/topics/installation/windows.html
Once you've installed the Salt minion you can start up the minion in a terminal with elevated privileges like this:
c:\salt\salt-minion.exe -l debug -c c:\salt\conf
You'll want to modify c:\salt\conf\minion and set the "master" option to be the IP or hostname of your master.