files are there, but when I run salt-cp to collect'em I get this funky error with no clear message. FWIW, I'm using different ports than the default:
publish_port: 44505
ret_port: 44506
...Files in place and server responding normally:
salt 'test1' cmd.run 'ls /tmp/test*'
test1:
/tmp/test1-1.json
/tmp/test1-2.json
/tmp/test1-3.json
...
Wonder what's wrong with that (also tried /tmp/test/ as dest):
salt-cp 'test1' /tmp/test* salt:// -l debug
[DEBUG ] Reading configuration from /etc/salt/master
[DEBUG ] Configuration file path: /etc/salt/master
[WARNING ] Insecure logging configuration detected! Sensitive data may be logged.
[DEBUG ] Reading configuration from /etc/salt/master
[DEBUG ] Missing configuration file: /root/.saltrc
[DEBUG ] MasterEvent PUB socket URI: /var/run/salt/master/master_event_pub.ipc
[DEBUG ] MasterEvent PULL socket URI: /var/run/salt/master/master_event_pull.ipc
[DEBUG ] Initializing new AsyncZeroMQReqChannel for ('/etc/salt/pki/master', 'XXX_master', 'tcp://127.0.0.1:44506', 'clear')
[DEBUG ] Initializing new IPCClient for path: /var/run/salt/master/master_event_pub.ipc
[DEBUG ] SaltReqTimeoutError, retrying. (1/3)
[DEBUG ] SaltReqTimeoutError, retrying. (2/3)
[DEBUG ] SaltReqTimeoutError, retrying. (3/3)
[ERROR ] An un-handled exception was caught by salt's global exception handler:
SaltClientError: Salt request timed out. The master is not responding. If this error persists after verifying the master is up, worker_threads may need to be increased.
Traceback (most recent call last):
File "/usr/bin/salt-cp", line 10, in <module>
salt_cp()
File "/usr/lib/python2.7/dist-packages/salt/scripts.py", line 359, in salt_cp
client.run()
File "/usr/lib/python2.7/dist-packages/salt/cli/cp.py", line 38, in run
cp_.run()
File "/usr/lib/python2.7/dist-packages/salt/cli/cp.py", line 105, in run
ret = local.cmd(*args)
File "/usr/lib/python2.7/dist-packages/salt/client/__init__.py", line 568, in cmd
**kwargs)
File "/usr/lib/python2.7/dist-packages/salt/client/__init__.py", line 317, in run_job
raise SaltClientError(general_exception)
SaltClientError: Salt request timed out. The master is not responding. If this error persists after verifying the master is up, worker_threads may need to be increased.
salt.exceptions.SaltClientError: Salt request timed out. The master is not responding. If this error persists after verifying the master is up, worker_threads may need to be increased.
Appreciate any help.
gosh dern it. From docs: Salt copy is only intended for use with small files (< 100KB). If you need to copy large files out to minions please use the cp.get_file function.
Sure... only < 100BK files.. very useful!
Related
I am searching for a long time on net. But no use. Please help or try to give some ideas how to achieve this.
my saltstack file code in github
saltstack file
install mysql salt code:
[root#salt_master srv]# cat salt/base/lnmp_yum/mysql/mysql_install.sls
repo_init:
file.managed:
- name: /etc/yum.repos.d/mysql-{{pillar['mysql_version']}}.repo
- source: salt://lnmp_yum/mysql/files/mysql-{{pillar['mysql_version']}}.repo
- user: root
- group: root
- mode: 644
mysql_install:
pkg.installed:
- names:
- mysql
- mysql-server
- mysql-devel
- require:
- file: repo_init
service.running:
- name: mysqld
- enable: True
after run cmd:
salt 'lnmp_base' state.sls lnmp_yum.mysql.mysql_install -l debug
always print log:
[DEBUG ] Checking whether jid 20170526144936867490 is still running
[DEBUG ] Initializing new AsyncZeroMQReqChannel for ('/etc/salt/pki/master', 'salt_master_master', 'tcp://127.0.0.1:4506', 'clear')
[DEBUG ] Passing on saltutil error. This may be an error in saltclient. 'retcode'
[DEBUG ] Checking whether jid 20170526144936867490 is still running
[DEBUG ] Initializing new AsyncZeroMQReqChannel for ('/etc/salt/pki/master', 'salt_master_master', 'tcp://127.0.0.1:4506', 'clear')
[DEBUG ] Passing on saltutil error. This may be an error in saltclient. 'retcode'
[DEBUG ] Checking whether jid 20170526144936867490 is still running
[DEBUG ] Initializing new AsyncZeroMQReqChannel for ('/etc/salt/pki/master', 'salt_master_master', 'tcp://127.0.0.1:4506', 'clear')
[DEBUG ] Passing on saltutil error. This may be an error in saltclient. 'retcode'
when i look salt node server, mysql already installed and start,but salt master server always print log, no exit.
I searched for days, but I could not solve it.
the same question when i install jboss.
Thanks in advance.
Two thoughts occur to me:
I think mysql has a basic configuration ncurses gui that requires user input to configure (Set default password etc). If I remember that correctly then your salt state is still running and waiting for a human to type at the screen. You can fix this by feeding it an answer/config file.
Stolen shamelesly from another post:
sudo debconf-set-selections <<< 'mysql-server-5.6 mysql-server/root_password password your_password'
sudo debconf-set-selections <<< 'mysql-server-5.6 mysql-server/root_password_again password your_password'
sudo apt-get -y install mysql-server-5.6
The other is that it may simply take longer than your salt timeout default for a task. That can be configured in salt at the salt cmd line with -t or the config file (forget which setting)
My /etc/salt/minion only has the line:
default_include: minion.d/*.conf
I wrote a very simple minion.d configuration:
cat /etc/salt/minion.d/saltminion
master: 123.123.xxx.xxx
id: test2
salt-minion -l debug
[DEBUG ] Reading configuration from /etc/salt/minion
[INFO ] Using cached minion ID from /etc/salt/minion_id: test1
[DEBUG ] Configuration file path: /etc/salt/minion
[INFO ] Setting up the Salt Minion "test1"
[DEBUG ] Created pidfile: /var/run/salt-minion.pid
[DEBUG ] Reading configuration from /etc/salt/minion
[DEBUG ] The `lspci` binary is not available on the system. GPU grains will not be available.
[DEBUG ] Attempting to authenticate with the Salt Master at 92.242.xxx.xxx
minion.d is completely ignored and I don't know where this 92.242.xxx.xxx ip is coming from, It looks like a major security hole to me.
Version: salt-minion 2014.1.13+ds-3
Would someone shed some lights here? please. Thanks
I trying to get the master to install software on the minion but it doesn't seem like its working. When I try a debug it looks like the problem is the following:
[DEBUG ] Missing configuration file: /root/.saltrc
Not sure how to go about fixing it. For more information my salt master version is 2015.5.10 (lithium) and my minion version is 2015.5.1. Also the complete debug log is:
[DEBUG ] Reading configuration from /etc/salt/master
[DEBUG ] Including configuration from '/etc/salt/master.d/master.conf'
[DEBUG ] Reading configuration from /etc/salt/master.d/master.conf
[DEBUG ] Using cached minion ID from /etc/salt/minion_id: ECS-141abdb2.ecs.ads.autodesk.com
[DEBUG ] Missing configuration file: /root/.saltrc
[DEBUG ] Configuration file path: /etc/salt/master
[WARNING ] Insecure logging configuration detected! Sensitive data may be logged.
[DEBUG ] Reading configuration from /etc/salt/master
[DEBUG ] Including configuration from '/etc/salt/master.d/master.conf'
[DEBUG ] Reading configuration from /etc/salt/master.d/master.conf
[DEBUG ] Using cached minion ID from /etc/salt/minion_id: ECS-141abdb2.ecs.ads.autodesk.com
[DEBUG ] Missing configuration file: /root/.saltrc
[DEBUG ] MasterEvent PUB socket URI: ipc:///var/run/salt/master/master_event_pub.ipc
[DEBUG ] MasterEvent PULL socket URI: ipc:///var/run/salt/master/master_event_pull.ipc
[DEBUG ] Sending event - data = {'_stamp': '2016-10-31T15:49:34.567058'}
[DEBUG ] LazyLoaded local_cache.get_load
[DEBUG ] get_iter_returns for jid 20161031084934590375 sent to set(['ds']) will timeout at 08:49:39.5975 53
[DEBUG ] jid 20161031084934590375 return from ds
[DEBUG ] LazyLoaded nested.output
Any help would be greatly appreciated since I'm new to this :)
Command used was:
sudo salt '<minion>' pkg.install 'chrome'
In case of windows you need to include some magic for this to work properly. Did you followed the instructions written here? A short summary of the docs content
# prepare your master to know about the win repos
salt-run winrepo.update_git_repos
# tell all connected windows minions to refresh
salt -G 'os:windows' pkg.refresh_db
# list available packages
salt -G 'os:windows' pkg.list_pkgs
# ask your minion for known chrome versions
salt '<minion id>' pkg.available_version chrome
# and finally install it; use version=x.x.x in case of a specific one.
salt '<minion id>' pkg.install 'chrome'
I am firing events from the salt minion using the command :
salt-call event.send 'kubemaster/kubernetes/started' '{master_ip:"10.102.28.170"}'
The master has reactor config:
reactor:
- kubemaster/kubernetes/started:
- /srv/reactor/testfile.sls
In the master debug mode I can see the event coming on the message bus but it does not show any message about rendering the sls file
[DEBUG ] Gathering reactors for tag kubemaster/kubernetes/started
[DEBUG ] Compiling reactions for tag kubemaster/kubernetes/started
[DEBUG ] LazyLoaded local_cache.prep_jid
Is there a better way to debug?
Is there any way to check if it is even looking for the sls file?
Thanks!
Could be a copy/paste error, but you are missing an indentation level for the second and third lines in the reactor config that you pasted.
I've been experimenting with Salt, and I've managed to lock up my highstate command. It's been running for hours now, even though there's nothing that warrants that kind of time.
The last change I made was to modify the service.watch state for nginx. It currently reads:
nginx:
pkg.installed:
- name: nginx
service:
- running
- enable: True
- restart: True
- watch:
- file: /etc/nginx/nginx.conf
- file: /etc/nginx/sites-available/default.conf
- pkg: nginx
The last change I made was to add the second file: argument to watch.
After letting it run all night, with no change in state, I subsequently Ctrl-C'd the process. The last output from sudo salt -v 'web*' state.highstate -l debug was:
[DEBUG ] Checking whether jid 20140403022217881027 is still running
[DEBUG ] get_returns for jid 20140403103702550977 sent to set(['web1.mysite.com']) will timeout at 10:37:04
[DEBUG ] jid 20140403103702550977 found all minions
Execution is still running on web1.mysite.com
^CExiting on Ctrl-C
This job's jid is:
20140403022217881027
The minions may not have all finished running and any remaining minions will return upon completion. To look up the return data for this job later run:
salt-run jobs.lookup_jid 20140403022217881027
Running it again immediately, I got this:
$ sudo salt -v 'web*' state.highstate -l debug
[DEBUG ] Reading configuration from /etc/salt/master
[DEBUG ] Missing configuration file: /home/eykd/.salt
[DEBUG ] Configuration file path: /etc/salt/master
[DEBUG ] Reading configuration from /etc/salt/master
[DEBUG ] Missing configuration file: /home/eykd/.salt
[DEBUG ] LocalClientEvent PUB socket URI: ipc:///var/run/salt/master/master_event_pub.ipc
[DEBUG ] LocalClientEvent PULL socket URI: ipc:///var/run/salt/master/master_event_pull.ipc
Executing job with jid 20140403103715454952
-------------------------------------------
[DEBUG ] Checking whether jid 20140403103715454952 is still running
[DEBUG ] get_returns for jid 20140403103720479720 sent to set(['web1.praycontinue.com']) will timeout at 10:37:22
[INFO ] jid 20140403103720479720 minions set(['web1.mysite.com']) did not return in time
[DEBUG ] Loaded no_out as virtual quiet
[DEBUG ] Loaded json_out as virtual json
[DEBUG ] Loaded yaml_out as virtual yaml
[DEBUG ] Loaded pprint_out as virtual pprint
web1.praycontinue.com:
Minion did not return
I then ran the same command, and received this:
$ sudo salt -v 'web*' state.highstate -l debug
[DEBUG ] Reading configuration from /etc/salt/master
[DEBUG ] Missing configuration file: /home/eykd/.salt
[DEBUG ] Configuration file path: /etc/salt/master
[DEBUG ] Reading configuration from /etc/salt/master
[DEBUG ] Missing configuration file: /home/eykd/.salt
[DEBUG ] LocalClientEvent PUB socket URI: ipc:///var/run/salt/master/master_event_pub.ipc
[DEBUG ] LocalClientEvent PULL socket URI: ipc:///var/run/salt/master/master_event_pull.ipc
Executing job with jid 20140403103729848942
-------------------------------------------
[DEBUG ] Loaded no_out as virtual quiet
[DEBUG ] Loaded json_out as virtual json
[DEBUG ] Loaded yaml_out as virtual yaml
[DEBUG ] Loaded pprint_out as virtual pprint
web1.mysite.com:
Data failed to compile:
----------
The function "state.highstate" is running as PID 4417 and was started at 2014, Apr 03 02:22:17.881027 with jid 20140403022217881027
There is no process running under PID 4417. Running sudo salt-run jobs.lookup_jid 20140403022217881027 displays nothing.
Unfortunately, I can't connect to the minion via ssh, as salt hasn't provisioned my authorized_keys yet. :\
So, to my question: what the heck is wrong, and how in the world do I find that out?
So, after a lot of debugging, this was a result of an improperly configured Nginx service. service nginx start was hanging, and thus so was salt-minion.
I had this happen when I aborted a run of state.highstate on the salt-master with Ctrl-C. It turned out that the PID referenced in the error message was actually the PID of a salt-minion process on the minion machine.
I was able to resolve the problem by restarted the salt-minion process on the minion, and then re-executing state.highstate on the master.