Error during include_vars when yml has embedded encrypted variables - encryption

Question is similar to Inline encrypted variable not JSON serializable
I have an Ansible configuration file that contains many plain text and a single encrypted value.
config_value: "something in plain text"
another_value: "another plain text value"
my_secret: !vault |
$ANSIBLE_VAULT;1.1;AES256
36343434346535636131316538313039386539363337326432336330393734306562336563386335
3663323065343330303039655530313833623439616630320a323730613534613338326263386136
33313033363737626230343432363764356336346363306466356330633561353831346433333366
3964386161306338300a333164333932333539333565303038643462643137663438326664336431
66613433396133633039385664366564626231313230383464356537636434646534
I encrypted the variable the following way, described here:
ansible-vault encrypt_string --vault-password-file path/to/password testvalue
When running include_vars I get the following error message:
TASK [include_vars] ************************************************************
task path: /var/jenkins_home/workspace/path/to/deploy-playbook.yml:21
[WARNING]: Failure using method (v2_runner_on_ok) in callback plugin
(<ansible.plugins.callback.default.CallbackModule object at 0x7f37a284fa90>):
u'testvalue' is not JSON serializable
Callback Exception:
File "/var/jenkins_home/workspace/path/to/virtualenv/local/lib/python2.7/site-packages/ansible/executor/task_queue_manager.py", line 375, in send_callback
method(*new_args, **kwargs)
File "/var/jenkins_home/workspace/path/to/virtualenv/local/lib/python2.7/site-packages/ansible/plugins/callback/default.py", line 100, in v2_runner_on_ok
msg += " => %s" % (self._dump_results(result._result),)
File "/var/jenkins_home/workspace/path/to/virtualenv/local/lib/python2.7/site-packages/ansible/plugins/callback/__init__.py", line 107, in _dump_results
return json.dumps(abridged_result, indent=indent, ensure_ascii=False, sort_keys=sort_keys)
File "/usr/lib/python2.7/json/__init__.py", line 251, in dumps
sort_keys=sort_keys, **kw).encode(obj)
File "/usr/lib/python2.7/json/encoder.py", line 209, in encode
chunks = list(chunks)
File "/usr/lib/python2.7/json/encoder.py", line 434, in _iterencode
for chunk in _iterencode_dict(o, _current_indent_level):
File "/usr/lib/python2.7/json/encoder.py", line 408, in _iterencode_dict
for chunk in chunks:
File "/usr/lib/python2.7/json/encoder.py", line 408, in _iterencode_dict
for chunk in chunks:
File "/usr/lib/python2.7/json/encoder.py", line 442, in _iterencode
o = _default(o)
File "/usr/lib/python2.7/json/encoder.py", line 184, in default
raise TypeError(repr(o) + " is not JSON serializable")
I've also tried the following commands to encrypt the secret but no luck:
ansible-vault encrypt_string --vault-password-file path/to/password 'testvalue'
ansible-vault encrypt_string --vault-password-file path/to/password "testvalue'
ansible-vault encrypt_string --vault-password-file path/to/password "'testvalue'"

The "error message" is actually not preventing the secret (my_secret) or any other configuration values in the same file from being usable.
I was on ansible 2.4.2 before, upgrading to 2.7.0 fixed the issue.

Related

Unhandled Exception for Downloader for Reddit

Having difficulty downloading a series of results when using the GUI. I'm able to log into my account, find a user and/or subreddit, but when I download I only get 1 result and it say
ERROR: Failed to extract due to: Unsupported Domain
Have downloaded all of the requirements in the requirements.txt file but not sure how to resolve this issue. Latest logfile result below:
"levelname": "CRITICAL",
"asctime": "02/12/2023 02:12:47 PM",
"filename": "main.py",
"module": "main",
"name": "DownloaderForReddit.main",
"funcName": "log_unhandled_exception",
"lineno": 48,
"message": "Unhandled exception",
"exc_info": "Traceback (most recent call last):\n File "/home/ads/projects/DownloaderForReddit-master/venv/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 2276, in _wrap_pool_connect\n return fn()\n File "/home/ads/projects/DownloaderForReddit-master/venv/lib/python3.8/site-packages/sqlalchemy/pool/base.py", line 363, in connect\n return _ConnectionFairy._checkout(self)\n File "/home/ads/projects/DownloaderForReddit-master/venv/lib/python3.8/site-packages/sqlalchemy/pool/base.py", line 773, in _checkout\n fairy = _ConnectionRecord.checkout(pool)\n File "/home/ads/projects/DownloaderForReddit-master/venv/lib/python3.8/site-packages/sqlalchemy/pool/base.py", line 492, in checkout\n rec = pool._do_get()\n File "/home/ads/projects/DownloaderForReddit-master/venv/lib/python3.8/site-packages/sqlalchemy/pool/impl.py", line 238, in _do_get\n return self._create_connection()\n File "/home/ads/projects/DownloaderForReddit-master/venv/lib/python3.8/site-packages/sqlalchemy/pool/base.py", line 308, in _create_connection\n return _ConnectionRecord(self)\n File "/home/ads/projects/DownloaderForReddit-master/venv/lib/python3.8/site-packages/sqlalchemy/pool/base.py", line 437, in init\n self.__connect(first_connect_check=True)\n File "/home/ads/projects/DownloaderForReddit-master/venv/lib/python3.8/site-packages/sqlalchemy/pool/base.py", line 652, in __connect\n connection = pool._invoke_creator(self)\n File "/home/ads/projects/DownloaderForReddit-master/venv/lib/python3.8/site-packages/sqlalchemy/engine/strategies.py", line 114, in connect\n return dialect.connect(*cargs, **cparams)\n File "/home/ads/projects/DownloaderForReddit-master/venv/lib/python3.8/site-packages/sqlalchemy/engine/default.py", line 489, in c###
I think it has something to do with the SQLite package or lack there of. I'm very new to coding and trying to run on my linux machine in a virtual environment.

Saltstack -> 'Pillar failed to render with the following messages'

On my FreeBSD I have a file packages.sls in the following path /usr/local/etc/salt/states
I'm getting the following error message when i do salt '*' state.apply packages :
freebsd:
Data failed to compile:
----------
Pillar failed to render with the following messages:
----------
Rendering SLS 'config' failed. Please see master log for details.
On the file master log i have the following details:
2022-06-02 10:05:12,222 [salt.roster :104 ][ERROR ][3425] Can't access roster for backend flat: Roster file "/usr/local/etc/salt/roster" not found
2022-06-02 10:05:12,434 [salt.pillar :900 ][CRITICAL][3427] Rendering SLS 'config' failed, render error:
found unexpected end of stream
Traceback (most recent call last):
File "/usr/local/lib/python3.8/site-packages/salt/renderers/yaml.py", line 62, in render
data = yamlloader.load(yaml_data, Loader=get_yaml_loader(argline))
File "/usr/local/lib/python3.8/site-packages/salt/utils/yamlloader.py", line 169, in load
return yaml.load(stream, Loader=Loader)
File "/usr/local/lib/python3.8/site-packages/yaml/__init__.py", line 114, in load
return loader.get_single_data()
File "/usr/local/lib/python3.8/site-packages/yaml/constructor.py", line 49, in get_single_data
node = self.get_single_node()
File "yaml/_yaml.pyx", line 707, in yaml._yaml.CParser.get_single_node
File "yaml/_yaml.pyx", line 725, in yaml._yaml.CParser._compose_document
File "yaml/_yaml.pyx", line 776, in yaml._yaml.CParser._compose_node
File "yaml/_yaml.pyx", line 890, in yaml._yaml.CParser._compose_mapping_node
File "yaml/_yaml.pyx", line 776, in yaml._yaml.CParser._compose_node
File "yaml/_yaml.pyx", line 892, in yaml._yaml.CParser._compose_mapping_node
File "yaml/_yaml.pyx", line 905, in yaml._yaml.CParser._parse_next_event
yaml.scanner.ScannerError: while scanning a quoted scalar
in "<unicode string>", line 3, column 27
found unexpected end of stream
in "<unicode string>", line 4, column 1
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/local/lib/python3.8/site-packages/salt/pillar/__init__.py", line 887, in render_pstate
state = compile_template(
File "/usr/local/lib/python3.8/site-packages/salt/template.py", line 99, in compile_template
ret = render(input_data, saltenv, sls, **render_kwargs)
File "/usr/local/lib/python3.8/site-packages/salt/loader/lazy.py", line 149, in __call__
return self.loader.run(run_func, *args, **kwargs)
File "/usr/local/lib/python3.8/site-packages/salt/loader/lazy.py", line 1201, in run
return self._last_context.run(self._run_as, _func_or_method, *args, **kwargs)
File "/usr/local/lib/python3.8/site-packages/salt/loader/lazy.py", line 1216, in _run_as
return _func_or_method(*args, **kwargs)
File "/usr/local/lib/python3.8/site-packages/salt/renderers/yaml.py", line 66, in render
raise SaltRenderError(err_type, line_num, exc.problem_mark.buffer)
salt.exceptions.SaltRenderError: found unexpected end of stream
2022-06-02 10:05:12,435 [salt.pillar :1224][CRITICAL][3427] Pillar render error: Rendering SLS 'config' failed. Please see master log for details.
My sls file packages.sls
install_bash:
pkg.installed:
- pkgs:
- bash
- vim
- curl
Any idea on how to solve this situation?
Thank you
It was a problem of DNS/Cache. Issue solved after changing the hostname in minion.id ,clear cache, accepted new key and restart.

Can't create new certificate with letsencrypt/nginx

I am trying to use Certbot to get new SSL cert for new domain, but I'm getting an error:
An unexpected error occurred:
UnicodeDecodeError: 'utf-8' codec can't decode byte 0x92 in position 715: invalid start byte
Please see the logfiles in /var/log/letsencrypt for more details.
This is the output from the letsencrypt log, not sure how to fix this.
2021-09-28 11:24:03,054:DEBUG:certbot.main:certbot version: 0.31.0
2021-09-28 11:24:03,055:DEBUG:certbot.main:Arguments: []
2021-09-28 11:24:03,056:DEBUG:certbot.main:Discovered plugins: PluginsRegistry(PluginEntryPoint#manual,PluginEntryPoint#nginx,PluginEntryPoint#null,PluginEntryPoint#standalone,PluginEntryPoint#webroot)
2021-09-28 11:24:03,064:DEBUG:certbot.log:Root logging level set at 20
2021-09-28 11:24:03,065:INFO:certbot.log:Saving debug log to /var/log/letsencrypt/letsencrypt.log
2021-09-28 11:24:03,066:DEBUG:certbot.plugins.selection:Requested authenticator None and installer None
2021-09-28 11:24:03,194:DEBUG:certbot.log:Exiting abnormally:
Traceback (most recent call last):
File "/usr/bin/certbot", line 11, in <module>
load_entry_point('certbot==0.31.0', 'console_scripts', 'certbot')()
File "/usr/lib/python3/dist-packages/certbot/main.py", line 1365, in main
return config.func(config, plugins)
File "/usr/lib/python3/dist-packages/certbot/main.py", line 1101, in run
installer, authenticator = plug_sel.choose_configurator_plugins(config, plugins, "run")
File "/usr/lib/python3/dist-packages/certbot/plugins/selection.py", line 225, in choose_configurator_plugins
authenticator = installer = pick_configurator(config, req_inst, plugins)
File "/usr/lib/python3/dist-packages/certbot/plugins/selection.py", line 25, in pick_configurator
(interfaces.IAuthenticator, interfaces.IInstaller))
File "/usr/lib/python3/dist-packages/certbot/plugins/selection.py", line 106, in pick_plugin
verified.prepare()
File "/usr/lib/python3/dist-packages/certbot/plugins/disco.py", line 251, in prepare
return [plugin_ep.prepare() for plugin_ep in six.itervalues(self._plugins)]
File "/usr/lib/python3/dist-packages/certbot/plugins/disco.py", line 251, in <listcomp>
return [plugin_ep.prepare() for plugin_ep in six.itervalues(self._plugins)]
File "/usr/lib/python3/dist-packages/certbot/plugins/disco.py", line 132, in prepare
self._initialized.prepare()
File "/usr/lib/python3/dist-packages/certbot_nginx/configurator.py", line 152, in prepare
self.parser = parser.NginxParser(self.conf('server-root'))
File "/usr/lib/python3/dist-packages/certbot_nginx/parser.py", line 38, in __init__
self.load()
File "/usr/lib/python3/dist-packages/certbot_nginx/parser.py", line 45, in load
self._parse_recursively(self.config_root)
File "/usr/lib/python3/dist-packages/certbot_nginx/parser.py", line 66, in _parse_recursively
self._parse_recursively(subentry[1])
File "/usr/lib/python3/dist-packages/certbot_nginx/parser.py", line 56, in _parse_recursively
trees = self._parse_files(filepath)
File "/usr/lib/python3/dist-packages/certbot_nginx/parser.py", line 207, in _parse_files
parsed = nginxparser.load(_file)
File "/usr/lib/python3/dist-packages/certbot_nginx/nginxparser.py", line 123, in load
return loads(_file.read())
File "/usr/lib/python3.6/codecs.py", line 321, in decode
(result, consumed) = self._buffer_decode(data, self.errors, final)
UnicodeDecodeError: 'utf-8' codec can't decode byte 0x92 in position 715: invalid start byte
2021-09-28 11:24:03,196:ERROR:certbot.log:An unexpected error occurred:
Based on the bug report from github, this is a unicode issue in configuration file.
Solution #1 - You need to generate an UTF-8 locale and launch certbot with it.
Like this:
LANG=en_US.UTF-8 certbot […]
Or, Solution #2 - replace unicode characters in nginx config
You can find the offending line with following command and delete it.
grep -r -P '[^\x00-\x7f]' /etc/apache2 /etc/letsencrypt /etc/nginx
And rerun certbot command.

Google Cloud Composer authentication for BigQueryHook

I'm trying to make use of the BigQueryHook but I am unable to get the service account authentication working.
I've followed the steps provided by Google and have copied the JSON file into the data/ directory in the environment's GCS bucket.
The Airflow Connection details have been filled in:
JSON Keyfile Path: /home/airflow/gcs/data/my-key-file.json
Keyfile JSON: content of the JSON file
Scopes: https://www.googleapis.com/auth/cloud-platform
Error in stackdriver:
Traceback (most recent call last): File "/usr/local/lib/airflow/airflow/models.py", line 374, in process_file m = imp.load_source(mod_name, filepath) File "/opt/python3.6/lib/python3.6/imp.py", line 172, in load_source module = _load(spec) File "", line 684, in _load File "", line 665, in _load_unlocked File "", line 678, in exec_module File "", line 219, in _call_with_frames_removed File "/home/airflow/gcs/dags/cloud_sql_to_bq.py", line 141, in df = get_config() File "/home/airflow/gcs/dags/cloud_sql_to_bq.py", line 71, in get_config bq_client = bigquery.Client(project=bq_hook._get_field("my-project"), credentials=bq_hook._get_credentials()) File "/usr/local/lib/airflow/airflow/contrib/hooks/gcp_api_base_hook.py", line 103, in _get_credentials key_path, scopes=scopes) File "/opt/python3.6/lib/python3.6/site-packages/google/oauth2/service_account.py", line 209, in from_service_account_file filename, require=['client_email', 'token_uri']) File "/opt/python3.6/lib/python3.6/site-packages/google/auth/_service_account_info.py", line 71, in from_filename with io.open(filename, 'r', encoding='utf-8') as json_file: FileNotFoundError: [Errno 2] No such file or directory: '/home/airflow/gcs/data/my-key-file.json'
Any idea why it can't see the JSON file?

Connect to web server using python3 socket

I'm running Python3 running Django on localhost and listening to http://127.0.0.1:8000/
and I want to test it by different ways like request and socket
import requests
res = requests.request("GET", "http://127.0.0.1:8000/")
txt = res.content
print(res)
res.close()
# <Response [200]>
It runs well, and
on my running Django console, it shows [07/Jul/2018 16:04:26] "GET / HTTP/1.1" 200 6744
Now instead requests to socket
import socket
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.connect(('127.0.0.1', 8000))
msg = b'GET / HTTP/1.1'
s.send(msg)
print(s)
s.close()
# <socket.socket fd=1324, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('127.0.0.1', 55312), raddr=('127.0.0.1', 8000)>
No error shows in my program
But I got the following exception on my running Django console
Invalid HTTP_HOST header: 'tony.jhou-1.xxservice.com:8000'. You may need to add 'tony.jhou-1.xxservice.com:8000' to ALLOWED_HOSTS.
[07/Jul/2018 16:08:10] "GET / HTTP/1.1" 400 63266
Traceback (most recent call last):
File "C:\Users\tony.jhou\AppData\Local\Continuum\Anaconda3\lib\wsgiref\handlers.py", line 138, in run
self.finish_response()
File "C:\Users\tony.jhou\AppData\Local\Continuum\Anaconda3\lib\wsgiref\handlers.py", line 180, in finish_response
self.write(data)
File "C:\Users\tony.jhou\AppData\Local\Continuum\Anaconda3\lib\wsgiref\handlers.py", line 274, in write
self.send_headers()
File "C:\Users\tony.jhou\AppData\Local\Continuum\Anaconda3\lib\wsgiref\handlers.py", line 332, in send_headers
self.send_preamble()
File "C:\Users\tony.jhou\AppData\Local\Continuum\Anaconda3\lib\wsgiref\handlers.py", line 255, in send_preamble
('Date: %s\r\n' % format_date_time(time.time())).encode('iso-8859-1')
File "C:\Users\tony.jhou\AppData\Local\Continuum\Anaconda3\lib\wsgiref\handlers.py", line 453, in _write
result = self.stdout.write(data)
File "C:\Users\tony.jhou\AppData\Local\Continuum\Anaconda3\lib\socketserver.py", line 775, in write
self._sock.sendall(b)
File "D:\emc\ve\lib\site-packages\gevent\_socket3.py", line 459, in sendall
return _socketcommon._sendall(self, data_memory, flags)
File "D:\emc\ve\lib\site-packages\gevent\_socketcommon.py", line 358, in _sendall
timeleft = __send_chunk(socket, chunk, flags, timeleft, end)
File "D:\emc\ve\lib\site-packages\gevent\_socketcommon.py", line 287, in __send_chunk
data_sent += socket.send(chunk, flags)
File "D:\emc\ve\lib\site-packages\gevent\_socket3.py", line 440, in send
return _socket.socket.send(self._sock, data, flags)
ConnectionAbortedError: [WinError 10053] 連線已被您主機上的軟體中止。
[07/Jul/2018 16:08:10] "GET / HTTP/1.1" 500 59
----------------------------------------
Exception happened during processing of request from ('127.0.0.1', 54587)
Traceback (most recent call last):
File "C:\Users\tony.jhou\AppData\Local\Continuum\Anaconda3\lib\wsgiref\handlers.py", line 138, in run
self.finish_response()
File "C:\Users\tony.jhou\AppData\Local\Continuum\Anaconda3\lib\wsgiref\handlers.py", line 180, in finish_response
self.write(data)
File "C:\Users\tony.jhou\AppData\Local\Continuum\Anaconda3\lib\wsgiref\handlers.py", line 274, in write
self.send_headers()
File "C:\Users\tony.jhou\AppData\Local\Continuum\Anaconda3\lib\wsgiref\handlers.py", line 332, in send_headers
self.send_preamble()
File "C:\Users\tony.jhou\AppData\Local\Continuum\Anaconda3\lib\wsgiref\handlers.py", line 255, in send_preamble
('Date: %s\r\n' % format_date_time(time.time())).encode('iso-8859-1')
File "C:\Users\tony.jhou\AppData\Local\Continuum\Anaconda3\lib\wsgiref\handlers.py", line 453, in _write
result = self.stdout.write(data)
File "C:\Users\tony.jhou\AppData\Local\Continuum\Anaconda3\lib\socketserver.py", line 775, in write
self._sock.sendall(b)
File "D:\emc\ve\lib\site-packages\gevent\_socket3.py", line 459, in sendall
return _socketcommon._sendall(self, data_memory, flags)
File "D:\emc\ve\lib\site-packages\gevent\_socketcommon.py", line 358, in _sendall
timeleft = __send_chunk(socket, chunk, flags, timeleft, end)
File "D:\emc\ve\lib\site-packages\gevent\_socketcommon.py", line 287, in __send_chunk
data_sent += socket.send(chunk, flags)
File "D:\emc\ve\lib\site-packages\gevent\_socket3.py", line 440, in send
return _socket.socket.send(self._sock, data, flags)
ConnectionAbortedError: [WinError 10053] 連線已被您主機上的軟體中止。
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "C:\Users\tony.jhou\AppData\Local\Continuum\Anaconda3\lib\wsgiref\handlers.py", line 141, in run
self.handle_error()
File "D:\emc\ve\lib\site-packages\django\core\servers\basehttp.py", line 88, in handle_error
super(ServerHandler, self).handle_error()
File "C:\Users\tony.jhou\AppData\Local\Continuum\Anaconda3\lib\wsgiref\handlers.py", line 368, in handle_error
self.finish_response()
File "C:\Users\tony.jhou\AppData\Local\Continuum\Anaconda3\lib\wsgiref\handlers.py", line 180, in finish_response
self.write(data)
File "C:\Users\tony.jhou\AppData\Local\Continuum\Anaconda3\lib\wsgiref\handlers.py", line 274, in write
self.send_headers()
File "C:\Users\tony.jhou\AppData\Local\Continuum\Anaconda3\lib\wsgiref\handlers.py", line 331, in send_headers
if not self.origin_server or self.client_is_modern():
File "C:\Users\tony.jhou\AppData\Local\Continuum\Anaconda3\lib\wsgiref\handlers.py", line 344, in client_is_modern
return self.environ['SERVER_PROTOCOL'].upper() != 'HTTP/0.9'
TypeError: 'NoneType' object is not subscriptable
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "C:\Users\tony.jhou\AppData\Local\Continuum\Anaconda3\lib\socketserver.py", line 639, in process_request_thread
self.finish_request(request, client_address)
File "C:\Users\tony.jhou\AppData\Local\Continuum\Anaconda3\lib\socketserver.py", line 361, in finish_request
self.RequestHandlerClass(request, client_address, self)
File "C:\Users\tony.jhou\AppData\Local\Continuum\Anaconda3\lib\socketserver.py", line 696, in __init__
self.handle()
File "D:\emc\ve\lib\site-packages\django\core\servers\basehttp.py", line 155, in handle
handler.run(self.server.get_app())
File "C:\Users\tony.jhou\AppData\Local\Continuum\Anaconda3\lib\wsgiref\handlers.py", line 144, in run
self.close()
File "C:\Users\tony.jhou\AppData\Local\Continuum\Anaconda3\lib\wsgiref\simple_server.py", line 35, in close
self.status.split(' ',1)[0], self.bytes_sent
AttributeError: 'NoneType' object has no attribute 'split'
I'm new to socket programming and just start learning networking, I think maybe I should send more request head. Please tell me how to do that, thanks.
Yes, you need to send more data. A minimal HTTP 1.1 request requires a second line that contains a Host header specifying the name or address of the host that should handle the request, and the request header lines must be followed by an empty line to tell the server that the header has been completely received.
In HTTP the end of a line is indicated by a pair of characters, a carriage return and a line feed. In Python strings \r represents a carriage return and \n represents a line feed. (The line feed character is sometimes called a newline character, hence the \n code.) This means that you have to write \r\n to mark the end of each line in the request.
So, what you need is:
msg = b'GET / HTTP/1.1\r\nHost: 127.0.0.1\r\n\r\n'
The final \r\n produces the empty line that indicates the end of the headers.
After you make that change it's likely that your server will report a different error, because your client program closes its socket immediately. That breaks the connection between the client and the server. When the server tries to send its response, it will find that the connection has been broken and it will probably complain about that. To avoid that, change your client to use s.recv to wait for and read the response from the server after it does s.send(). Something like:
while True:
resp = s.recv(8000)
if resp:
print resp
else:
break

Resources